Mini PC vs Server for running virtual machines

[@scsirob]

Thanks for the review. I went through a similar exercize end of last year and found a really good candidate. It's the Topton CW56-58 which runs an AMD 5825U 8-core /16-thread laptop CPU with low TDP. I bought three of these, William Lam reviewed this micro-PC a while later. They have a number of advantages over the GMKtek unit: - 8 core instead of 4 - Supports up to 64GB of SO-DIMM memory (not DDR-5 though) - Has 4 (four!) builtin Intel i226V 2.5GbE NICs, no need for USB NICs - Has two M.2 NVMe slots - Runs VMWare ESXi 8 right out of the box, ESXi 7 with a network fling. The only thing that's not functional on these gems is the TDP 2.0 module (same as on the GMKtek). Other than that I run ~40 VMs on two of these and I have a unit to spare for tests. These units are somewhat more expensive but you'd need to buy two or three GMKteks to match the Topton. Adding a QNAP TSB464 all-NVMe shared storage, a no-name pfSense firewall/router, a NetGear MS510TXM storage switch and a NetGear GS324T access swich, my homelab consumes ~130W.

[@VirtualizationHowto]

@scsirob, that is a very interesting unit for sure. I saw William's post on the unit. Will definitely have to check it out. Have you ran vSAN on the units? I know William didn't mention it, and wondering if you had a chance to test?

[@scsirob]

@@VirtualizationHowto Yes, I ran a 2-node vSAN on these for a short while when toying with vSphere 8 eval. I installed a 128GB and a 2TB SSD in both units. vSAN ran just fine with a witness node on a different machine. Managing a 2-node vSAN isn't much fun and eats a lot of resources, so when I installed vSphere 7 for my final lab deployment, I reverted back to shared storage on the QNAP.

[@VirtualizationHowto]

@scsirob, so you were able to get it up to 128 GB memory? I was thinking they are showing 64 max, but great to know if so. I have seen a lot of these little mini PCs showing they were limited to 64.

[@scsirob]

@@VirtualizationHowto No, not 128GB main memory unfortunately! The unit has 2 SO-DIMM slots and there are no DDR4 SO-DIMMs larger than 32GB that I'm aware of. So 64MB main memory per server is the max on this one as well. vSAN takes a fair chunk of that 64GB, hence my decision to go back to shared storage. Sorry for being unclear. What I meant is that I installed a 128GB NVMe SSD as boot/system disk for ESXi, and a 2TB NVMe SSD for vSAN in each server. There are two M.2 slots in each server.

[@LucasSousaRosa]

@@scsirob If so, do you work with the 3 mini-pcs together? Do they work as a single server or is each one accessed separately?

[@charr6108]

A really good spec for mini pc would be something like i9 12900h 64gb DDR4 ram 8TB M.2 NVMe SSD :) Not so much for power efficiency but its indeed powerful and small

[@mackayd]

Interesting discussion but having been in the game long enough to go full circle. Grew tired of compatibility so realised the simplest solution is high core count desktop ddr4 is cheap and comes in large dimms on windows with VMware workstation and nested esxi with vcenter native to workstation. Consoles accessible directly on desktop. With 16 core and 128gb ram on nvme storage allows for 2-3 64,GB ram esxi hosts. I've run 2 DC solution with 2 esxi hosts per vcenter. 20-30 vms. It allows me to rapidly create customer environments with little hassle and I don't need to mess with switches etc. Lab refreshes are simply a case of buying new desktop parts or whole device

[@VirtualizationHowto]

Drew, that is awesome and a great perspective. What type of workstation are you running? Is it off the shelf hardware or did you build it yourself? What are your specs? Thanks again!

[@mackayd]

@@VirtualizationHowto Its a few years old now so might be due a refresh. CPU Ryzen 9 3950x 128GB DDR4 with 3 NVME PCI4 M.2 which originally i RAID0'd; the benchmarks 12GB/s were great but didn't translate to real world gains and with added risks of running RAID0 it wasn't worth it. I didn't go all out on hardware as it was kinda a PoC but when i refresh it i think i'll probably look at stepping it up to a threadripper as it removes the RAM limitation.....but having just checked the prices of even the older gen ones i'll probably stick with Ryzen9 chip. Either way at the refresh point you end up with two desktops and as long as you have a dual nic motherboard you could create a tagged private network between them. Oh the only real limitation is a software limit imposed by VMware. All vNICs in VMware workstation are limited to around 1-1.5Gb/s even where there the vNIC is backed by 10G interfaces and VMXnet3 drivers are used. It seems to be a software limitation but after a few fruitless deep dives i've never found a setting in an ini file to remove it,.

[@budgiekiller2]

Many older network and storage devices are no longer supported because VMKlinux drivers have been completely removed from vSphere 7 ESXi . I have not found any 2.5gb ethernet for native ESXi-v7 yet. 10gb is cheaper these days so I don't mind. You can still lab v7 and v8 ESXi as VMs in WMware Workstation (on windows, linux or Mac) 2.5 gb networks.

[@VirtualizationHowto]

@budgiekiller2 thank you for your comment! Yeah definitely a bummer on the drivers.

[@nadtz]

The GMK's are nice with that tiny form factor but I prefer any of the options with intel 2.5GbE nics which aren't hard to find for 'mini servers' that are slightly larger. I hope someone fills the 10GbE hole one day, right now the cheapest/smallest option I could find is an asrock epyc board which is kind of hard to find and isn't actually all that cheap by the time you add memory nor as small as the typical mini pc. Not quite as low power either though it's not bad for what it is. I just got a beelink ser 5 which will end up being my windows machine but since it will probably be a while before I move my main desktop to Linux it will be a fun box to play around with till then.

[@VirtualizationHowto]

@nadtz definitely understood....I would have liked to have had 2.5 Gbps on this unit, would have made it almost perfect for what it is at this price and form factor.

[@PowerShellWizard]

I have a ryzen 7 5th Gen mini PC with 32gb of RAM and nvme storage. VMs run like total crap on it with tons of lag even with power settings cranked to performance and running windows server with hyperV. Long story short : don't waste your time or money. Not designed for the purpose

[@VirtualizationHowto]

@droidiscrap which model and make do you have. What problems do you see specifically? Would love to have more insights from you

[@DudeSkinnyTall]

So, could you please elaborate on the exact model of that Realtek-based USB NIC? That's very interesting quirk from vmware)

[@budgiekiller2]

I believe many of the Realtek nics were still relied on the old VMKlinux drivers which are no longer supported in vSphere 7/8 Realtek have to build drivers for their cheaper chipsets that use the ESXi native drivers APIs like the other manufacturers adopted years ago.

[@chrisumali9841]

thanks for the demo and info, have a great day

[@VirtualizationHowto]

Thanks, you too!

[@-rm-rf]

Enjoyed this one! would be cool with an actual “tier” list based on some criteria’s you already mentioned in the vid. Power consumption, value/price, form factor (size), features, also noice levels would be interesting to know, both on idle and when you push em. At that price point for the one in the vid, I could for sure consider picking up a couple just to play around with cluster features and not having to wait in my next actual build or use nested proxmox to get out HA and/or CEPH as a concept

[@oneamongmany1392]

Cool video! I bought something with the same spec recently (n100, 12gb, 1tb) - smaller than that but only a single nic (but built in WiFi). Put Proxmox on it because I saw realtek NIC and value my sanity. I can manage it over WiFi and use the NIC for bridge. Certainly cool to play around with but would you use this for a service that your home relies on (e.g. for a firewall or NAS) ? That's a little different to the fun/labbing scenario you described. The appeal is it's tiny and can run 24/7 so it may as well right?

[@hotrodhunk7389]

I'd run a router on almost any reliable bare metal. I'm not going to virtualize my router though... don't need the Internet to go down if I have to do work on the server.

[@marcusrodriguesadv]

I have 3 Mini PCs ina Proxmox Cluster with Ceph Storage, i use a Beelink EQ Pro, N305 with 16gb of RAM, it uses 2280 nvme nd tehre is room for a SATA, i boot the servers form an externam 256gb nvme in a case USBC 3.2 and use the internal storage slots with 2TB NVME and 2 TB SATA in each node, as a Ceph storage, they have 2 nics of 2.5gbps and i use them connected to a 2.5gbps switch to sync data and conected by cable to my wi-fi 6 router... this is my homelab...

[@VirtualizationHowto]

@marcusrodriguesadv, that is awesome! Thanks for sharing that. Do you see any performance issues with 2.5 Gbps as opposed to 10 Gbps? I am wondering myself on this front. Let me know if you have any notes to share on that. Thanks again Marcus!

[@marcusrodriguesadv]

@@VirtualizationHowto i didn't use 10gbps yet, still expensive in Brazil, 2.5gbps is my fastest, the rj45 ports in the wi-fi router are 1gbps.

[@frdurso]

@@marcusrodriguesadv I also live in Brazil... and Marcus is right, it is just too expensive around here, I use 1 GK3 Pro - Celeron N5105/ 16Gb RAM and 512Gb NVME running Proxmox and an old raspberry 3B+ running pi-hole... for Storage a VERY OLD DLINK DNS-320L with 2x2TB Sata Hard Drives everything in 1Gbps speed... We can only dream with your setup around here 🥲.... Maybe I will buy a decent NAS in a couple of months...

[@ronwatkins5775]

Question. How did you get vcenter? Isn't it cost prohibitive? I've asked VMware and it's several thousand dollars.

[@VirtualizationHowto]

@ronwatkins5775 I can't remember if I had asked if you signed up on the VHT forums? Be sure to sign up there and we can discuss further: www.virtualizationhowto.com/community

[@HelloHelloXD]

We live in the future! Great times

[@accesser]

I'm currenting using an older 8th gen intel NUC to run ESXI, I'd love another soon, I'd really love somebody to create a lab orientated NUC with lights out management, that would be really quite fun

[@VirtualizationHowto]

@ccesser that is awesome, are you stil sticking with the NUC for an additional node? I am with you, would be so amazing to see a vendor create something geared towards home lab enthusiasts for sure

[@accesser]

@@VirtualizationHowto I'd happily pay a little extra for a proper built in KVM hardware I understand maintaining the software costs development money so some kind of once off (reasonably priced) licensing might also be okay

[@silopolis-yt]

​@@VirtualizationHowtoAm I wrong to think that the home lab enthusiasts wishes somehow meet the edge needs?

[@normtomlins3821]

I was looking at the mini PCs but you could buy an Acer TC-1760 with a Core i5-12400 for about $450CAD load them with 64Gb's of memory and add a 10Gig Nic card and you have a nice little lab setup for ESXi 8.0. I purchased 3 of them.

[@alexm7043]

How is esxi able to run on that processor?

[@normtomlins3821]

@@alexm7043It doesn't have the E Cores and works with just the P-CORES. The on board NIC card won't work...

[@TheBeardedLibertarian]

I’ve been running a unraid server on a mini for some time. Nice to see a micro doing it 😊

[@nectarinetangerineorange]

Or maybe a 16c/32t ryzen with 32GB DDR5 and a 2.5 Gbe NIC (and if oyu want to get a little hack-y, oyu can pull the wifi card out and replace it with another dual 2.5Gbe NIC... run one nic direct to your data server for hostsing your vm ssds on a massive encrpyted, compressed zfs array & use the internal nvme drive as a OS drive and a giant swap/tmpfs space to allow the OS to keep active things in the ARC and passive things on local nvme storage, with everything backstopped by the ZFS server.... and use the second nic as a virtual switch to connect all the vms to the actual network...) Just a thought....

[@Pongo8844]

Can you show how well the unit at doing real work instead of idling VMs?

[@VirtualizationHowto]

@Pongo8844, thank you for the comment. In most home labs, the VMs are mainly going to be idling, with a few peaks here and there. Would be interesting to do a few benchmarks for sure though with some real and simulated load going on.

[@eduardonobrega77]

Can you share how you installed ESX in the Mini PC, i have one with a ryzen 8 cores and 64GB of memory with proxmox to do some tests but i would like to test with VMware too. I've tried to use the usb fling but i had problems with the NVME too on the install - Thanks

[@Mongo11b]

Are the sodimm slots soldered or can you upgrade them at all?

[@VirtualizationHowto]

I ^believe^ on the GMKtek they are soldered in there...when you remove the undercarriage, you only have access to the hard drive. So I think you are stuck with the RAM you purchase with on this unit...

[@Mongo11b]

@@VirtualizationHowto ewww. Still a cool nano machine but soldered ram is such a buzzkill

[@dnldnl4880]

Hard to find mini pc with 16 or 32 Gb ram and nics that work out of box

[@derrysan]

How good it is for 24/7 workload?

[@KCYT2010]

With the buyout of VMware I can't imagine too many home lab builders being interested in running ESXi .

[@medlotfi4720]

I'm not sure why you're not thinking to post some serious videos on what we can do inside those vms (Windows servers or Linux ) and some Ha vMotion StorageHA some serious stuff

[@VirtualizationHowto]

@mediotfi4720 definitely hoping to get a series of videos going on deeper home lab dives

[@ronm6585]

Thank you.

[@VirtualizationHowto]

You're welcome!

[@JasonsLabVideos]

First

[@JP-lz3vk]

Wrong. Nice try. Better luck next time.

[@VirtualizationHowto]

Thank you @jasonslabvideos!

[@JasonsLabVideos]

I was first @@JP-lz3vk

[@shephusted2714]

they are terrible - just build a pc for your intended purpose and save some dosh plus learn a few things - they lack a lot like expansion room, pci slots, upgrade path, more ram slots etc