250 Virtual Machines on a Proxmox Mini PC

[@dave24-73]

Great for schools and home labs. What this demonstrates to me is how stable Proxmox was.

[@AlonsoVPR]

Great Experiment man, I wonder how many VMs can a N100 handle with 16gb

[@itguy8541]

This experiment is very interesting, you pushed the limit, I think it is already possible to replace VMs in small companies, real enterprise environment, the cost is much lower, as long as a disaster recovery plan is applied including backups and clones using mini PCs. The investment and maintenance will still be much cheaper than the current servers. Thanks for the video.😊

[@dave24-73]

Most large companies wouldn’t consider these, but it would be great for schools.

[@isLife-if8lz]

Very interesting experiment! You could also use Terraform or Ansible to automate the template cloning, must have been tiring doing the full clone 250 times!

[@seanwoods1526]

Thanks for putting this together. I agree the new mini PCs are amazing the only thing I miss is my ilo ports from my servers.

[@armandosreis]

Great test. I would have like to see the power drawn from the system and also the temperatures it rose to. Those can have a negative impact in the life expectancy of your hardware, so please be aware of this if you try to consolidate your entire homelab into a miniPC :)

[@ewenchan1239]

It REALLY depends on what your VMs/LXC containers are doing though. It's one thing to be able to SPIN UP 250 VMs. It's another thing to be able to automatically run 250 instances of prime or stress-ng or something that's either memory, CPU, and/or iGPU intensive. Have each of the 250 VMs open up 250 Chrome tabs and let's see how well it will or won't be able to handle that. 250 VMs, with most of them being idle, doesn't really demonstrate much.

[@wentropia]

I guess the main point is 250 x 2gb = 500 gb... and you make it in a 96GB equipament because they have same content.

[@ewenchan1239]

​@@wentropia "I guess the main point is 250 x 2gb = 500 gb... and you make it in a 96GB equipament because they have same content." Well...yes and no. Given that he cloned the VMs, therefore; it relies HEAVILY on the Kernel Shared Memory (KSM) for "memory" "compression". However, as the VMs start doing ACTUALLY different things, where you aren't able to take advantage of KSM, then you will MASSIVELY oversubscribe the CPU and the RAM. When you oversubscribe the CPU, one of two things will happen: 1) Either it will slow everything down as it competes for CPU time/resources or 2) Linux will crash where it will print messages to the console (sometimes, behind the scenes) where it will say something to the effect of "CPU stuck. CPU clock x behind real clock by y ticks." (Something like that.) Basically, there isn't enough clock cycles to process what the VM needs it to process, so it "lags" behind. And if the VMs lags behind enough, it will start crashing/shutting down altogether. (Ask me how I know this.) When you oversubscribe the RAM, these two things will happen: 1) It will start swapping like made until the swap file size limit is attained, and then you will put pressure on the CPU as it tries to swap the swap. (I've had this happen with my OASLOA Mini PC and also my Qnap TS-453Be NAS, where it hit a load average of 46 on a 4-core CPU, because of this.) 2) If it can't swap, then it will just throw an "out of memory" error, and shut down/crash. (Again, ask me how I know this.) The better way to test this would be that he sets up each of the VMs such that when it starts up, it immediately loads a RAM and/or CPU intensive workload (e.g. solving the linear algebra problem: Ax=b using the Gauss elimination with partial pivoting), and then test/see how many VMs he's able to clone and start, whilst it is set up to immediately run that workload. 250 VMs that's sitting at idle, doing nothing is precisely that: it is and means nothing. Compare and contrast that with 10 VMs that's all running the Gauss elimination with partial pivoting solver, where the size of the square matrix is designed to fit within the 2 GB RAM limit/footprint, that would be significantly more telling as to what the system can ACTUALLY do, in terms of ACTUAL work. Not as clickbait-y of a title for a video, but significantly more accurate in terms of what the system can ACTUALLY handle.

[@robertlijkendijk4497]

Dear lord, did you really create 250 VM clones manually? That must have been quite an undertaking… Interesting video! Very cool to see this amount of VMs running on such a small box, and how proxmox deals with this amount of memory overcommit.

[@markmonroe7330]

Excellent presentation. Thank you. Really enjoyed this one. Even more impressive since that CPU only has 8 actual cores with hyperthreading. I think this test really shows how good both AMD processors are and also Proxmox.

[@JamesGreen-gv4yn]

Nice! I'm curious what the Proxmox log was showing when it shut down those VMs on you. It was off the bottom of the screen in your recording. Then again, you may even have had that closed. But that would provide more details on the VMs that got shut down and possibly the exact reason why they were shut down.

[@KifKroker]

my guess is that the journal would have shown oom kills, expected in this scenario

[@JamesGreen-gv4yn]

You made a comment about the IO Delay going up. Would like to see a video about this specifically. Back when I had Proxmox servers that had regular old hard disk drives with spinning platters, I don't remember ever seeing this type of behavior. It was only with SSDs that I started to notice it. It also appears that the type of SSD can make a huge difference in this and many posts I've read indicate that an "Enterprise grade" SSD is required for Proxmox. But I haven't found a good definition or explanation for what an Enterprise grade SSD means.

[@kingneutron1]

Enterprise grade usually means it has PLP (power less protection = supercapacitor) and a high TBW rating, you don't want to use e.g. a 256GB QLC EVO for proxmox bc it will die in less than a year

[@AdHdEntertainmentLLC]

This is awesome. Given my struggles of running two VM's on my daily driver laptop with 32 gigs of ram on a 4core gaming laptop with a 1650 Nvidia graphics card. Makes me really hope I can win the minisforum mini PC. As I'm trying to move from my current govt non IT role I really need something with more power!! Can't believe you had 250 VM's running. I don't need that many.

[@pbrigham]

This thing has more processor power than any server from 10 years ago....

[@SB-qm5wg]

I think you're right.

[@phasechange5053]

10? more like 5-7... 10 years ago servers were like 8core max per cpu. I recently deployed a 5K $ 16 core server thats what intel is demanding and it was supermicro so on the cheap side.

[@crissdell]

What i dor get it, is how which 8 cores can you virtualize that amount of VMs?

[@joeykennedyloa2567]

@@phasechange5053 not really, while the 7940hs should still be faster, 10 years ago xeon e5s went up to 16 cores(xeon e5 2698 v3). 7 years ago the first intel xeon scalable cpus launched and i'm pretty sure the 28-core top models are a good bit faster than the 7940hs. Though price and energy cost will be way higher of course...

[@davidfarrell1062]

My homelab is exaclty as you describe. A Dell mini PC with 2 x 2TB m.2 and 64gb of DDR4 with 8 VMs and backup to an external drive (Not PBS yet). Its exactly what I need and have spare with no m.2 or ram in case I have hardware issue. Could bring it online but the cost and power consumption (even though 10w on idle). These are very capable but what I see is there are very few youtube channels that focus on these setups. They mostly focus on labs or use clusters and NAS and multiple devices in a rack etc. Most dont need or have that but find it hard to get material for managing a single proxmox nuc. Be great to see more of that type of content.

[@marcoschirrmeister]

Very strange experiment. As a next step let each VM actually do something with its 2GB memory. Otherwise, what's the point?

[@Crystawth]

You could probably push it a bit further if you expand the swap space. sitting at 8GB fully used in the example you gave. But personally, I try to avoid any swap usage if possible since disks are so much slower.

[@peteradshead2383]

I like you to try running 250 windows 11 machines next and see how they run. My proxmox is a bit of a mixed bag where I try to run them as a lxc if I can , but sometime it's just easier to even linux machine to run as a VM. I would say running real life Linux apps like , SMB server , frigate , jellyfin , pi-hole , databases and graphs etc , about 1 meg minimum is more realistic , on my system it's ZFS which is grabbing most of the memory for cache .

[@rafaelg8238]

great, congrats video. a doubt: do you created lxc/vms manually or automatically? is possible to create dynamically automated any lxc/vms? if yes, how?

[@amorpheuses1627]

I had the same question. There's no way I'd do this manually. I see that proxmox has helper scripts - but does this gui have the equivalent of an api one could use?

[@ZenRebel33]

Where are the VMs you are spinning up stored? Are you using storage from a large drive in the Mini PC or are you using network attached storage?

[@lukastram4990]

You see that long stick, i assume its an Nvme SSD

[@Eklund-yw6og]

​@@lukastram4990 Says Samsung 980 PRO 2Tb on the label.

[@chetan_naik]

Next you can run a test where you can test just how many VMs you can run on that machine with everyone one of the VMs running stress test.

[@andyk9685]

You are crazy, I hope you know that. That's why I love your channel!

[@Alexander-kf5tc]

Great! But, are those vm's payloaded with some software? Like lamp for example? Or other?

[@KifKroker]

ofc not but lamp would only consume resources when they get requests... ofc you are not gonna run 250 working lamp stacks on a single mini-pc

[@milonzed9392]

Hello, great demonstration there, i've got nothing more to add except one thing : the test was a little skewed, because as your video shows, there was 100% swap used all along. I don't know Proxmox, i'm more an ESX guy, but your hypervisor consumes RAM, as well as your VMs. Using 8 GB swap is *almost* similar as adding 8GB more RAM. Albeit SSDs / NVMEs are a much slower and wearable kind of RAM. We could imagine a scenario where you would use 1TB of NVME, and be able to run thousands of VMs, althrough as i've said earlier, much slower. 8GB is not a lot, but it's significant on a 96GB RAM system (12,5%). But i DO agree with you, there never was a better time to run a homelab, even if i despair seeing more memory limits and connectivity on these mini-PCs ! I personnaly prefer the "shuttle" size, the shoebox format, for heat evacuation, dual 10Gbps fiber connectivity, and the 128GB RAM limit).

[@chrisumali9841]

Thanks for the demo and info. This is great, and makes me wonder if I should scrap my power hungry HP DL 360 G8. Have a great day

[@johnscabintech]

Could you do the same test but using the MS-01 would be good to compare Intel vs AMD plus you could compare the eco threads and its impact in the home lab?

[@kingneutron1]

Keep up the good work.I suspect the number of actual productive VMs on your current setup would be around the 25-or-less mark, would be interesting to script a test on a 2nd PC that pulls webpages and DB data from random VMs on the proxmox server in parallel. Responsiveness is also going to be limited by the SSD and whether you are mirroring or not

[@crissdell]

Why 25?

[@kingneutron1]

@@crissdell Wild-assed guess based on number of cpu cores /3 or some chit ;-) Only good way to tell is to test your worst-case scenario with everything-on and stressing the system at once with CPU + I/O. If it's too-slow for the intended users / use-case then you scale it back and add more servers

[@kingneutron1]

@@crissdell I came up in mainframe-land so my expectations may be a bit different. We ran those things 100% CPU 24/7 to get value for money. When I say "actual productive VMs" it means the max number of simultaneous things-running that still gives acceptable response-time for an interactive user. Only way I know of to do that is to start with a somewhat-reasonable number of VMs based on available cores and run parallel stress tests. If it gets too slow then you scale it back and add more servers (or possibly mirror your storage)

[@crissdell]

@@kingneutron1 but how? If the physical machine only has 8 physical cores, I mean 1 cores is equal to 1vcore?

[@agrep]

What about containers instead of VMs ? Thanks for this video.

[@LeeMyers-Jr]

I would like to see another version of this, instead of Ubuntu use Windows VMs.

[@dosomething6975]

You could get maybe a 5th of the VMs if your lucky. Windows is heavy 😅

[@dougrohde3598]

Brandon, Which small mini pc would you go with? MS-01 or MS-A1 or something else?

[@ronm6585]

Thanks for sharing. 👍🏻

[@timothywcrane]

Its a nit-pick but with 100% swap usage how much would that really hit performance on anything prod? That aside, this is a true testament to how far we have come in applied SFF. Thanks.

[@GundamExia88]

Great video. But isn't it a little misleading when you use 8GB of swap? If I were to test, I would see how many or how well it can run with 0 swap, then 1GB, then 8GB swap. Using swap is a very useful way to extend the RAM because it provides the necessary additional memory when the RAM space has been exhausted and a process has to be continued. It is especially recommended when you have less than 1Gb of RAM. Although in the end, everything depends on you. With no swap, when your RAM gets full and the Linux kernel has no space to write into, your system will crash. On the other hand, if you have a swap space, it will be used by the Linux kernel and your system will keep working, though much slower. I assume your swap and VM are on NVME, if they were on SATA SSD, your disk io delay might get hampered further? As for some VM powered off, have you checked the logs? I'm sure those VM experience OOM. But it's great video because I'm running a lot of VM and LXC containers on my minisforum MS-01 and HP Elitedesk G4 and both uses less power than my 1U-4U servers. MiniPCs runs much better and more versatile than rpi. Keep up the good work.

[@dustind9242]

Interesting that this autoplayed... I was literally on amazon/ebay shopping for additional ram to give an old motherboard cpu combo (asus hero vi ryzen 7 1800x) a new lease on life as a proxmox server. May just stick with the 16gb that I have and just send it..

[@ericandrews4861]

That’s a great test to perform. With all of them running, but no workloads being down on those VM‘s, it makes sense that you should be able to do that. However, if you have workloads going on any of those virtual machines, you absolutely will not be able to put that money on there.how many can you spend up the actually have viable workload on them. That would be interested in seeing

[@crissdell]

How can he run so many VMs, isnt there cores limitations?

[@ivanmaglica264]

Is there a reason why you chose Full clone vs. Linked clone? On NVMe and being used as temporary workload, linked clone would make more sense...

[@Monarchias]

I am thinking if we give proxmox in a custom system a dedicated nvme swap drive with the same amount of swap space as the amount of ram. Could that be a help for spinning up more or similar amounts with more workloads in each VM? We learnt that these days we don't need that much of swap space because of speeds we achieved. But we are the next level, so swap can be useful again with these fast hard drives?

[@SB-qm5wg]

That Mini PC has more horsepower than my 4U homelab server

[@haydenc2742]

Cool and all...but impractical test...they are idle, they are using that magical balancing thing...won't be the same for multiple type OS VM's or ones that are being heavily utilized. Interesting test though Keep em coming!!!!

[@etienne4403]

What happens when you increase the Swap? Or is that not possible? I wonder how many stressed machines you can have before it comes crashing down.

[@zerlegen3919]

Hello. Just want to be sure about something. The Amazon page says the Geekom mini computer comes pre-installed with 32GB of dual-channel DDR5 5600MT/s RAM (expandable up to 64GB). Expandable to 64 GB. You are saying you put 96 GB in it and it worked without issue?

[@lukastram4990]

Nice Test, but those Boxes have too less Connectivity options, give it 10 or 4x2.5 Gbps Ethernet etc

[@cybersamurai99]

Nice video! - 250 VMs but without any load at all, I mean?. How is that a test ? (obs: how is that comment being a hater its obisous LOL)

[@TheMrDrMs]

If only there would be these mini pc's with sfp+ or if I could be greedy two, 1 for my fault domain/block storage and 1 for my network access. That's okay, I enjoy my enterprise eqpt but hate my power bill

[@kingneutron1]

check servethehome , minisforum has a mini-pc with 2x 10Gbit SFP+ and so does Qotom

[@jeroboam4486]

You didn't script the cloning of the VM?

[@AI-xe5jr]

Imagine having this in the 80s😭

[@kazamihazaki8467]

wait 96 GB Ram? but i just visit the official website its just up to 64 GB Ram

[@chetan_naik]

It looks as though each VM was just a process.

[@userslinx6865]

That a wrong number or "bythical"/mythical number!!??? It should be 2^8 so 255 so 256 VM (because counting from 0...to 255). Or the software might at put a limit at 16bits?? But we are on a 64bits system so it might be it 2^64???????

[@ikorbln]

This is a linear test for cpu/memory. I think when you realy use them random, the shared i/o and the shared network a real problem. The Host is also at 100% swap usage, also not good.

[@et_phonehome_2822]

I got to learn Proxmox instead of using VirtualBox.

[@kingneutron1]

Yes. With proxmox and Linux VMs / CTRs you can get actual 10Gbit network speeds; virtualbox will not come anywhere near that.

[@niky1016]

"Hello Brandon, what configuration did you use to have 96 GB of RAM if the specifications of the equipment indicate that it only supports up to 64 GB?"

[@niky1016]

Hello Brandon, I read your article and there you explain how you reached 96 Gb. Thanks for the information.

[@thulanintini5311]

HI , Can l run eve ng ccie lab on this?

[@lse123polis]

Ram of 96GB is 64GB + 32GB... modules .... ?

[@techadsr]

I think he said 2 48G modules.

[@lse123polis]

​@@techadsr Yes, but RAM Modules capacity is the power of 2, that's: 1 2 4 8 16 32 64 128 256 --- never ever heard of 48 ---is not power of 2

[@jaylenharris2891]

@@lse123polismax capacity for soddr5 stick is 48gb

[@michaelmcconnell7302]

i dont get it

[@RossCanpolat]

Now try without swap

[@twigahappy]

The randomly shutting down VM's comment makes me wonder if my current cloud provider is over provisioning!

[@Htbaa]

Ha that’s an interesting one. I’ve got several VPS’ at one provider and it’s always my mail server that suddenly reboots. There’s never anything in my log files indicating what the heck happened.

[@RomanHaussener]

OK cool, but why?

[@kjakobsen]

Now try Windows Server 2022

[@matthiashavrez]

So, are they haters or are they correct ?

[@2008spoonman]

Why not use proxmox helper scripts for creating vm’s? Manually creating is a bitch…

[@hardi.stones]

Well, I'm not impressed. Most of these VMs are idle! Yes, I was expecting ALL VMs run the stress test utility or some program(s) that simulate user activity. Now that would be realistic.

[@cls-pf2sx]

keep in mind that what all these hyper visor tech is actually doing for you... it allows you to use the idle CPU cycles that the h/w is wasting on waiting for the human/other slow input devices to the system. so CPU stress test within the individual VM is basically a stress test on the host CPU, so if you want to 'run' more VMs in such use case, get a bigger/better CPU on the host. I don't believe it is the purpose of this video to show how great the CPU is.

[@gajcia1931]

wtf, use ansible for it xd

[@the_13east]

stop using light mode jesus