ESXi and Intel VT-x support on Up Squared N4200 8GB ?

Osman Demirtas

I am considering buying up/up2 boards to create a virtualised hadoop cluster, but beforehand I have got a couple questions that are awaiting an answer. I'd appreciate any information from excisting up2 users or devs.

[li]Does the UP2 N4200 8GB bios support Intel Virtualization Technology (VT-x) ?[/li]
[li]Wil Vmware ESXi work on the UP2 N4200 8GB and can I run 2 Ubuntu linux VMs on this machine ?[/li]
[li]Does ESXi support eMMC memory and can I allocate it to a VM?[/li]

The VM setup I had in mind running on 1 Up2 N4200 8GB 128GB eMMC + 128GB M2 SSD is as follows:
1 x 2 Core 4GB 128GB eMMC running Ubuntu + Hadoop on ESXi
1 x 2 Core 4GB 128GB M2 SSD running Ubuntu + Hadoop on ESXi

eduncan911

upfantr wrote:

[li]Does the UP2 N4200 8GB bios support Intel Virtualization Technology (VT-x) ?[/li]

Yes.

upfantr wrote:

[li]Wil Vmware ESXi work on the UP2 N4200 8GB and can I run 2 Ubuntu linux VMs on this machine ?[/li]

I couldn't get it to boot under EFI. ESXi says it supports EFI, but there are various posts on their support site about it not working correctly.

The UP boards are all UEFI only - no Legacy BIOS booting options.

upfantr wrote:

[li]Does ESXi support eMMC memory and can I allocate it to a VM?[/li]

As stated above, I couldn't get my USB to boot with EFI (3 different images on 2 different sticks).

But I wouldn't let this stop you. You can use the onboard SATA or mPCIe/mSATA ports to add additional storage. This is what I did for pfSense on the UP2.

upfantr wrote:

The VM setup I had in mind running on 1 Up2 N4200 8GB 128GB eMMC + 128GB M2 SSD is as follows:
1 x 2 Core 4GB 128GB eMMC running Ubuntu + Hadoop on ESXi
1 x 2 Core 4GB 128GB M2 SSD running Ubuntu + Hadoop on ESXi

Personally, I am running Xen on ArchLinux to minimize memory usage. I have 1 Arch VM running a Docker Swarm node and 1 VyOS VM I am experimenting as my new internet gateway.

I am currently working on isolating openvpn, dnsmasq, and other network services each in their own docker container to minimize risk.


You may always want to review this page where I've tested several operating systems as well:

https://up-community.org/wiki/Operating_Systems

Osman Demirtas

@eduncan911, Thanks for your respond.
Also I'd like to know if anyone have tried the Samsung 850 EVO mSATA 250GB 3d v-nand (TLC) SATA-600 on the Up2 and did it work well ?

eduncan911

There are 100s of mSATA drives available. Asking if someone specifically tested one exact model is rare.

You shouldn't have a problem with that one, or really any mSATA drive. I tested a 480GB Mushkin in my UP Squared. So if it supports that, I'd say it supports just about any mSATA drive.

Upp3r

eduncan911 wrote:

This is what I did for pfSense on the UP2.

are you running your pfsense 2.4 without any problems?
running baremetal or virtual?

how did you solve this issue at boot with the sdhci/pci errors? (see attachment)

eduncan911

Upp3r wrote:

eduncan911 wrote:

This is what I did for pfSense on the UP2.

are you running your pfsense 2.4 without any problems?
running baremetal or virtual?

how did you solve this issue at boot with the sdhci/pci errors? (see attachment)

I was running bare metal, yes. You disable the onboard eMMC to get rid of the sdhci/pci errors since FreeBSD 11 (what pfSense 2.4 is based on) doesn't support/has a bug in for Intel's eMMC 5.0 driver. Instead, use an mSATA or SATA device on the UP Squared (or even an 8GB USB stick).

For pfSense and the limitations of FreeBSD 11, refer to this thread: https://up-community.org/forum/general-discussion-up2/1814-how-to-install,-other-then-ubuntu,-pfsense-freebsd#6413

Once disabling eMMC, I used a SATADOM module (there are some pics in another pfSense thread around here somewhere).

I demonstrated pfSense running native on the UP Squared in the UP Booth at the World's Maker Faire in NYC 2017. In the demo I showed it sending and receiving ~850 Mbps bi-directional over both NICs, with the CPU around 8% usage.

I also did a quick download test of AES256 CBC OpenSSL encryption via OpenVPN using another UP Board to test and got around 450 Mbps. That's extremely impressive with Intel's AES-NI hardware acceleration.

Since pfSense 2.4 is based on FreeBSD 11, and FreeBSD has a bug that for Intel's eMMC 5.0 driver, I installed it on the SATADOM module connected that extended up on the onboard SATA port. I disabled the eMMC functionality in the BIOS to get it installed (since FreeBSD gave a lot of timeouts and took a long time to boot).

I don't recommend the SATADOM modules. They are expensive/overpriced for their size, and do NOT fit any UP2 chassis as they are too tall! (really meant for 1U servers)





Personally though, running pfSense on the UP² hardware is like driving 5 minutes through downtown NYC traffic in a Lamborghini just to get to your apartment. It's really overkill and you can do a lot more with this beast.

Hence, why I've hacked together an ArchLinux + Xen install and playing with some VMs. Once I get the process down solid, I'll write a tutorial.

Upp3r

thanks for getting back to this
i've been following the other thread as well.
so i also disabled the internal emmc and connected an old 240 GB SSD via SATA and installed pfsense 2.4 on it.
but i still get those anoying errors and boot up takes because of that about 5 mins.
did i miss something in my BIOS settings?

to come back to topic
i totally agree this beast can do more. so my wish-setup would be run a esxi and 2-3 vm's on it (i have the n4200 with 2 GB memory), but for my purpose this should be good. i'd like to have a pfsense vm, a pi-hole vm and maybe a reverse proxy as a vm.
i've also been looking into proxmox but i've been using esxi for the past so it would be my goto strategy.

your archlinux +xen thing sounds interesting, would be nice to hear more once it's done.

at the end of the day i just want a flexible solution where i can backup vm's, do snapshots and play arround with the stuff without having troubles to brick it

eduncan911

Most likely yes, missing a few other options. Make sure to disable all 4 options in that eMMC area in the BIOS.


Sounds like you are where I was about 5 to 6 years ago.

I'm now concerned mostly about security as my kids grow up and start using devices more.

So my primary focus is designing a hands-off device that will self-update every 24 hours with the latest security and version releases. This is Arch Linux, but it's not for the faint of heart.

I really like Pi-Hole as well, especially the data it provides so I can graph it in Grafana. But I want more up-to-date software as Pi-Hole doesn't update that often. Therefore, the solution is to run my own dnsmasq in docker, and run my own adblocker script and http servelet in another docker to keep them isolated. I can generate almost the same data as Pi-Hole does on Grafana.

Upp3r

awesome - the trick with the 4 options helped. thank you. works like a charm now

i feel you. set it up, never touch and some times just monitor it an have a look at some graphs
that's why i like pfsense. i used pfblockerng on pfsense, but for some reason i ran across pihole and since then i have it running on a vm on an esxi
i will definitly look into pfblockerng once the new release will be out. much more nicer and more customizable. unfortunatley theres no ETA yet.

so for the topic, have you yet set up an ESXi on the upboard? if yes, anything to be aware of?

eduncan911

I keep this current with the list of operating systems I test:

https://up-community.org/wiki/Operating_Systems

GavinLu

I am trying to install ESXi 6.7 on Up Board now, and it is partly succeeded except eMMC support. Please go to Up BIOS board and check my question. Thanks,

eduncan911

I'm going to copy-n-paste my comment to another thread here for posterity.

To be clear, here are the list of operating systems supported by UP:

https://wiki.up-community.org/Operating_Systems

I have added my results in the past for other OSes at the bottom (need to update Arch and Xen on ther as they are fully functional with the latest kernels).

OS support basically comes down to two things:

• Intel eMMC 5.0 storage controller, released in 2012 (yeah, that old)
• 1 Gbps Realtek RTL8111G network controllers (both the UP Board and UP Squared use the same)

The UP Core also has an Intel WiFi/BT module as well.

The ApolloLake and CherryTrail Intel chipsets have also been out for several years as well.

In my experience, most newer OSes have these in the last 3 years or so.

It is the older and slower-to-release kernels that have problems with the eMMC 5.0 Intel controller. Even though it was released in 2012, BSD-based OS (FreeBSD, pfSense) and CentOS-based OS (e.g. XenServer) have improper and buggy implementations of this controller (I linked to the FreeBSD bug for it elsewhere). And therefore, will not recognize properly nor install onto the built in storage. For the UP^2, you have the options of using either a SATA or mSATA drive to install onto. However, the UP Board has neither. You're looking at a USB drive install.

This is not an UP problem as the 6-year old controller is native to the Intel chipsets; however, it is an upstream problem with the OS as it's a well known chipset and old storage controller by now. User's who want this must make their voices known in those communities to fix their implementations. Newer Linux kernels and Windows 10 has support them for many years now.

Which brings me to ESXi... We all know how picky vSphere is with their approved hardware list - which is an Enterprise product. However, their free ESXi product has even less support and usually runs even further behind in hardware support.

CyBHFal

For anyone who wants to have virtual machines on an up squared board, you can use XCP-ng

XCP-ng 8 use a newer kernel than XenServer 7.x and now support eMMC 5.0 Intel controller.

But, to install XCP-ng 8.0 on internal eMMC, I needed to patch the installer on my usb stick prior to install because the installer did not detect eMMC and had a partition layout which did not fit with my 32Gb eMMc.

After that, it works perfectly to host several little VMs depending of your RAM size.

Here are the little changes to apply:

https://github.com/CyBHFal/host-installer/compare/master...CyBHFal:Tiny-master

From 1e169f7dc240f0c6a89de88870d265701030246a Mon Sep 17 00:00:00 2001
From: CyBHFal <cybhfal@free.fr>
Date: Sun, 1 Sep 2019 14:58:18 +0200
Subject: [PATCH 1/2] Allow installation on eMMC

---
disktools.py | 2 +-
diskutil.py  | 3 +++
2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/disktools.py b/disktools.py
index 15c9a70..2554eb0 100644
--- a/disktools.py
+++ b/disktools.py
@@ -488,7 +488,7 @@ def diskDevice(partitionDevice):

 def determineMidfix(device):
     DISK_PREFIX = '/dev/'
-    P_STYLE_DISKS = [ 'cciss', 'ida', 'rd', 'sg', 'i2o', 'amiraid', 'iseries', 'emd', 'carmel', 'mapper/', 'nvme', 'md' ]
+    P_STYLE_DISKS = [ 'cciss', 'ida', 'rd', 'sg', 'i2o', 'amiraid', 'iseries', 'emd', 'carmel', 'mapper/', 'nvme', 'md', 'mmcblk' ]
     PART_STYLE_DISKS = [ 'disk/by-id' ]

     for key in P_STYLE_DISKS:
diff --git a/diskutil.py b/diskutil.py
index 8d0aa70..0e6fc40 100644
--- a/diskutil.py
+++ b/diskutil.py
@@ -129,6 +129,9 @@ def mpath_disable():
 # /dev/md    : md has major 9: each device has 15 minors)
 disk_nodes += [ (9, x * 16) for x in range(16) ]

+# /dev/mmcblk : emmc has major 179
+disk_nodes += [ (179, x * 16) for x in range(16) ] 
+
 def getDiskList():
     # read the partition tables:
     parts = open("/proc/partitions")

From 004f4d7335a984b0f067f554f0e694a37b04d95a Mon Sep 17 00:00:00 2001
From: CyBHFal <cybhfal@free.fr>
Date: Sun, 1 Sep 2019 15:00:05 +0200
Subject: [PATCH 2/2] Tiny installation on 32Gb eMMC

---
 constants.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/constants.py b/constants.py
index 70ed0de..80e552d 100644
--- a/constants.py
+++ b/constants.py
@@ -90,7 +90,7 @@ def error_string(error, logname, with_hd):
 # reports the total system ram after the Xen heap.  The UI should
 # display the value given by MIN_SYSTEM_RAM_MB_RAW.
 min_primary_disk_size_old = 12 #GB
-min_primary_disk_size = 46 #GB
+min_primary_disk_size = 12 #GB
 max_primary_disk_size_dos = 2047 #GB
 MIN_SYSTEM_RAM_MB_RAW = 1024 # MB
 MIN_SYSTEM_RAM_MB = MIN_SYSTEM_RAM_MB_RAW - 100
@@ -101,16 +101,16 @@ def error_string(error, logname, with_hd):
 # filesystems and partitions (sizes in MB):
 boot_size = 512
 root_mbr_size_old = 4096
-root_mbr_size = 18432
+root_mbr_size = 4096
 root_gpt_size_old = 3584
-root_gpt_size = 17920
+root_gpt_size = 3584
 root_size_old = max(root_mbr_size_old, root_gpt_size_old)  # used for free space calculations
 root_size = max(root_mbr_size, root_gpt_size)  # used for free space calculations
 backup_size_old = 4096
-backup_size = 18432
+backup_size = 4096
 swap_file_size = 512
 swap_size = 1024
-logs_size = 4096
+logs_size = 1024
 logs_free_space = 20

 # filesystems and partitions types: