Running VMs in TrueNAS Scale - Should you run this instead of Proxmox?

[@privateger]

In short: No. Long: NooOoOOOoOoooOooo!

[@silentbyte33]

Hahahahahah!!!

[@rumblpak]

This is what I came to say. Truenas is great for storage but awful at just about everything else.

[@rob.granger]

Twenty One Long Minutes Later... :) Ok Twenty Long Minutes the two on beer was useful.

[@0xKruzr]

A+; no notes.

[@VeronicaExplains]

I ran my homelab rig on Scale for a week. It was the longest month of my life. I wish them well and hope they can solve it, but I've stuck with Debian+KVM for good reason. Everything just works for me there.

[@KyleMahaney]

When was this? Were the issues mostly revolving around the older apps system they're now replacing with proper docker in the next release? I don't have any pcie passthrough for my vm's, but I do have a ton of things running and its been great. It even runs my ansi text environment linux shell host for retro terminals lol.

[@VeronicaExplains]

@@KyleMahaney Timeline was about a year ago. And I didn't use their app system, I mainly run VMs. At the time I had no need to test containers.

[@brockwilkie6022]

@@VeronicaExplains my impression has been. TrueNas for storage and docker apps, and the occasional VM work. Pretty much the same with UNRAID. But serious VM work and containers should be proxmox, or other hypervisor, with VM running docker apps. ONly have the basics myself but that is just what I keep seeing from people.

[@jttech44]

Debian/Qemu/KVM is about as bulletproof as it comes to be honest. You can also do that without proxmox at all, if you're cool with managing everything via CLI (or some other, inferior gui). That said, you could probably Python your way to a decent enough gui without much trouble if you're just looking to do some specific things.

[@aquinamedia4508]

Was curious on the topic, ended up running TrueNAS on Proxmox, have to admit it feels a bit iffy should things go south. First try and all, both on VM and NAS 🤣

[@TheJulianJES]

VLAN support for VMs is there since some years. You create a VLAN in the network section and then attach that to the VM in NIC settings. It works fine for me. noVNC was fully replaced by Spice due to security issues in the VNC protocol, but the usability is still bad. However, using virt-viewer should work much better compared to the web client. TrueNAS 24.10-RC.2 is already out and the full release of 24.10 will be out by the end of the month. There are multiple fixes in that version. For example: The Spice VM password is correctly hidden when entering it on 24.10. The 8 length password limitation issue is also resolved. Also, for the disk latency test, was the virtual disk passed through via AHCI or VirtIO? 24.10 might also improve that, but I haven't tested it. Just a guess.

[@Felix-ve9hs]

9:23 he used AHCI, that's probably the issue here

[@3k3k3]

So.. time for a new and updated Proxmox guide? :) EDIT: It should be noted that i am a huge fan of Jeffs guides, i have spend many hours learning and installing stuff, that i don't really need, but just for the sake of it. I mean, nerdy and all. But it's been a while since he took our hand and walked us through a complete Proxmox guide, with window install, Truenas Setup, GPU pass-through on a consumer card, backup etc etc.. Yeah i know, months of content/work.. sorry Jeff

[@thetalesofdaneandco]

Please

[@paulnolastname9422]

I second this motion.

[@equizzix4804]

I third this motion

[@BlitzFingers]

Yeah, all the people leaving VMware need help!

[@dragonhawk1492]

*In Darth Sideous' voice:* DO IT.

[@keyem4504]

I am running 2 Proxmox machines and was looking for a way to run a Proxmox Backup Server and a NAS. When I got an older Server for free I decided to run TrueNas bare metal and virtualize PBS on top of it, since the NAS performance was more critical to me than that of the backup server. It works, but TrueNas is nowhere near Proxmox as a Hypervisor. I felt restricted in a lot of things I wanted to do. So, for me Proxmox is my go-to for hypervisors and TrueNas Scale my favorite NAS solution. Both great in their area of expertise but no competition for each other.

[@patrickburyk9973]

That's the way I'm running things now and was hoping to migrate my Proxmox VMs over to TrueNas Scale. This video (and the se comments) have pretty much convinced me not to change anything as everything is working reliably as it is. Thanks for all the confirmation!!!!!

[@greenprotag]

You can pry Proxmox from my cold dead fingers LMAO

[@greenprotag]

I also have found some uses for Hyper-V depending on hardware and Hardware needs

[@moogs]

Truth

[@gh8447]

Challenge accepted.

[@autohmae]

It has a GPLv3 Affero license, be sure to set up a mirror of the git repos, just in case. While the license is a great choice to keep it open, BUT the project does demand a Contributor License Agreement, which is a big red flag.

[@shabadabadoo4326]

For VM's it's the beesknees.... but storage and dockers are cantankerous AF.

[@kenzieduckmoo]

"Trying to outsmart the room" is the modus operandi of this entire channel :P

[@Momi_V]

I've been running TrueNas Scale as my sole Homeserver/NAS/Hypervisor for over half a year now. It has worked perfectly fine for running a Reverse Proxy VM, my Nextcloud VM (+ related services), a VM for Jellyfin and Handbrake (with GPU passthrough) along with the occasional Test VM for random stuff i come across. I've standardized on OpenSUSE MicroOS (which is extremely easy to provision in a wide variety of different hosting environments) and the only thing I found lacking were the janky VM Display viewer and Snapshots not including the VM configuration. I tested both Proxmox and XCP-ng for about 6 months each before (and even some more exotic solutions like a Hyperconverged Ceph cluster based on RHEL derivatives) and switched over to Scale, because I just didn't need the extra features. For me the better functionality and UI for handeling Storage (Block device and ZFS options), User management and Share Permissions along with the Performance and Efficiency increase, as well as the complexity decrease of not messing around with a virtualized NAS appliance was worth the usability tradeoff in the Virtualization department.

[@bw3388]

I agree. I just built a new home server with the latest release of TrueNAS Scale and I'm perfectly fine with the results. I have a few containers running, and a couple linux VMs... nothing super complicated, but it all just works. Yes, the Spice VRT sucks and is the one thing I'd change... nothing is perfect. I looked at Proxmox & xcp-ng and favored the latter, but in the end, wanted my NAS OS to have direct PCIx access to all drives without question. Yes, you can virtualize and pass the HBA through, but many experts including TrueNAS state to not virtualize for production use cases. TrueNAS Scale is not the best pick for enterprise solutions IMO. But for a small, single, do-everything home server... I have no complaints. If I was doing this at work, sure, I'd buy 3-6 hosts, break it all apart with clusters, etc... but for home where energy consumption is factor - no thanks.

[@ianhuu]

I also have truenas as my sole homeserver. it runs 4 vm 1 homeassistaint 2 win 11 vm to remote in and mess around when i am at work 3 xubuntu for the same reason 4 ubuntu server runin docker instances (after wendell's level1tech guide), bc truenas built in docker solution is barbone, for my 20+ docker image setup. i soent 10 minutes to create the vm-s i never touched that part of truenas since, it runs for me falwlessly

[@airman_85uk]

I'm running TrueNas Scale and I'm running a basic debian VM as an application host. It's only running docker with a dozen containers. Works fine for that!

[@gioriv]

Same here, I’m running 2 VMs one for Docker and another for WordPress. Runs smoothly!

[@sparksnmagic8695]

I would be really interested if you could retry this experiment with the Electric Eel version of TrueNAS.

[@dustojnikhummer]

I was hoping he used EE, it had some noticeable (according to patch notes) improvements

[@fuzzbreezie81]

I really wanted to see the as well now that its in the RC stage. I think that its a good place to start as home lab for testing as long as its not mission critical services.

[@andymammel]

I am running EE, Spice is still hot garbage.

[@retrogear]

Unfortunately, still a mess on the UI side... rsync module is also broken when trying to use as a Synology backup target - works fine in 24.04.2.3

[@PeterBuffon]

In my opinion, if you run truenas bare metal and run vms in it it's like using an ISP router that is wifi/switch/toaster, in my opinion you only run truenas for a NAS software. And a hypervisor as a hypervisor.And you can run truenas as a VM too.

[@TeflonBilly426]

The problem with that is that many systems these days leave a lot of power and utility unused in that scenario.

[@dustojnikhummer]

Assuming you can pass thru your HDD controller yes. TrueNAS is a very basic hypervisor but it is there.

[@ahuesphoto]

Thank you Peter! One should use the right tool for the job...

[@Dsturzy]

I am running truenas scale on bare metal with a win10 vm that has been rock solid for at least a year. Nothing crazy, but I can’t complain. It does the one small task I need it to do

[@regularguy1934]

Once I tried running Truenas Scale as you, because in the store I could install docker containers such us jellyfin, and Radarr... File permissions were too confusing and I ended installing a VM instead of using those apps

[@RyanDSM1975]

Trooper is an awesome all around pub food beer. Goes great with bar pizza, tacos, fish & chips, the whole gamut. Just an easy drinking, smooth, British lager as you said.

[@farawaythrower]

The timing of this video is impeccable, thanks!

[@jonathanhaney9235]

Pretty much nailed it. TrueNAS is fine as a storage host and not much else. I pretty much do what you suggest. Proxmox host, TrueNAS VM and other VMs doing fun things inside proxmox.

[@MorkaiAU]

I immediately saw the Iron Maiden "Trooper" beer on the desk and cannot sufficiently express my enthusiasm for the band. Saw them recently in Melbourne and tried the beer at a pop up bar here near the venue too. Also recently got a "Powerslave" tattoo on my forearm which has been on the wishlist for a long time.

[@mrgwei]

All of these problems are exactly why im keeping truenas strictly as a NAS and switching all hypervisor tasks over to proxmox.

[@ravenssettle]

I have a proxmox cluster for my compute and a truenas server that's just for storage. I've considered running some docker containers in truenas once that feature is released, but never did any VMs there.

[@ravenssettle]

Should I think about virtualizing truenas and putting proxmox on it? 🤔🤔

[@eliasalcazar6554]

@@ravenssettle I've done that to run some LXC's before, and to help make sure I'll always have a quorum incase any of my other nodes go down. Never had any problems running it that way. It ran fine with 128GB of storage and 8GB of RAM.

[@jttech44]

That was the previously recommended way to do it actually, truenas for pooled, shared storage, proxmox for compute. Nowadays you lose so little by virtualizing that it makes sense to just keep it all under one roof. Also, it's very likely you can just throw proxmox on your truenas host instead, and then pass through your HBA's to a truenas VM and import your pools and be back in action with very little nonsense. You can probably even get away with backing up your truenas config and just importing it, assuming the versions match.

[@eliasalcazar6554]

@@jttech44 You can 100% flip-flop from a VM to bare metal install by donwloading/uploading the config file! I've done it a few times in my testing and TrueNAS is none the wiser to the changes.

[@nwallace]

I use truenas scale to host my caddy based reverse proxy VM. It is very simple and I can rebuild it in just a few minutes if I have a problem. I would never host any serious VM on truenas. The hypervisor feature is just not finished and not trustworthy. I have had much better success with the apps feature. I am super excited to have that switch to docker in this next release which will make it even easier to use for me.

[@musiqtee]

Yep, I’ll stay with TrueNAS running inside ProxMox - doing just the NAS/ZFS stuff it’s really good at. All other tasks in other PM containers or VMs. That has been running close to four years now, shutting it down only for thunderstorms… 😅

[@Fradler40]

Timely video. Just set up a TNScale and had the exact same experience. I thought I'd just made all the mistakes possible: I'm now vindicated. As a NAS, works great. I'll keep my Proxmox for VMs for now. Will see what the next update holds with reworking containerization.

[@motmontheinternet]

You finding bugs in TrueNAS is just normal. I've never been able to use TrueNAS's GUI for an afternoon without running into something showstopping. It's also confusing because they explicitly recommend you use the GUI and not the console.

[@jttech44]

They recommend you use the GUI, until they don't, and then the documentation is... occasionally sparse. The situation really is that they don't recommend CLI for modifying settings that are in the GUI, for obvious reasons tbh, but if you never use the GUI, that's not going to be a problem either.

[@wojtek-33]

​@@jttech44It will be. Some things you change in the CLI will be reverted on reboot.

[@classicrockonly]

Yep. I had to drop to the command line to turn a single disk zpool into a mirrored zpool. So dumb. It's an easy job, but I would expect it to be an available task in the GUI

[@idan678]

how about open media vault?

[@DaveAndrus]

And all this time I thought it was just my inexperience! I gave up on VM's in TrueNAS because I assumed I was using a mature system with a proven wizard, and I was just too dense to figure it out. Thanks for clarifying the situation, and at the same time making me feel a little less like an idiot!

[@MikeButash]

I saw this hoping you'd say TrueNAS Scale was great, but apparently not! Thank you for the details, as your complaints, particularly around networking and pci pass-through, are directly applicable to my use cases too. I have a supermicro nas box I was weighing using proxmox and loading truenas into as I do this with my firewall today, but wasn't sure how pci pass-through would behave for passing the sata controller for the 4x spinners on it. I saw this thinking maybe I wouldn't have to, but as you said proxmox base with truenas in vm sounds like the way to go. Thanks again!

[@nickspacemonkey]

I've found myself not using a VM these days. The jailmaker script works really well for making systemd-nspawn containers and you can choose from a pretty good list of LXC images.

[@Yuriel1981]

Been running Truenas Scale since Angelfish, It has been a learning experience thats for sure. While, yes yes you can attach a VLAN to a VM as others have mentioned here. It is not super intuitive in any capacity. I have also experienced most of the issues you did as well especially with SPICE VNC. But I have successfully gotten my Homeassistant OS Supervised in a stable VM setup (Not the recommended installation method btw). I am running most of my services through the apps available in their catalog though. Jellyfin, Octoprint, Frigate, OLlama, Pi Hole and Tailscale. All stable and working pretty well. Edit: btw Im a welder, I dont code. Just a tech nerd and have most of the wonderful Truenas community to blame? for my success. Lol

[@glentheAussie007]

Great video Jeff, always informative and helpful, I would like to see you do a video on CEPH, on setting it up, running it, etc, and also your thoughts of course. Cheers keep up the great videos.

[@Austin-xx2nu]

I currently have a TrueNAS scale server acting as the KVM at a church I attend. They have very basic needs when it comes to a hypervisor, but it is working and has not really had any issues.

[@jason-budney7624]

Thanks for suffering so i don't have to, Jeff! I'll be like you and stick with Proxmox while using TrueNAS in a VM. What a beer to pick for when this video went live. Maiden played at the Moda last night, what a great show!

[@lolish1234]

I've been using it temporarily but once you are used to all the features/functions of esx/proxmox then this feels like stone age :D Better to run truenas as a vm with hdd passthrough and having a proper hypervisor.

[@marcosv8426]

How hard is it to do ZFS disk storage/configuration/permissions directly in Proxmox without setting up a truenas VM? Hate to have to allocate all those resources to a VM for something so basic to ZFS.

[@lolish1234]

@@marcosv8426 honestly don't know since i've never yet used proxmox, only vsphere/esx but am going to delve in it this fall when I get a new server

[@david_sanchez]

Hey Jeff. Great video, as usual! I figured I'd chime in with my thoughts. Albeit, in a personal-use/small software dev env. TrueNAS's virtualization really is clunky. It's not refined and can be confusing with how some things are implemented (i.e. over-provisioning memory & such). I've also experienced most of the things you mention in the video. However, I really only run Linux virtuals to simulate a fail-safe database environment for software development purposes. That being said, VNC really isn't a thing for me. It's all pretty much SSH and if I need direct console access, I just use the Serial Shell utility in the VM's drop-down. I also haven't experienced too much gripe with GPU pass-through. The steps to set that up aren't very intuitive but once you do it, it's done. I can say that once you do get it all working and it's doing its thing, you really don't ever have to touch it again. This isn't to say that "it just works" because it really is finicky but, I haven't had any issues with it since I initially set it up. All this to say... TrueNAS is storage-centric. Proxmox is virtualization-centric. If TrueNAS wants to realistically compete with Proxmox's hypervisor capabilities and features, iX would most likely need to release a product with virtualization as its core. When people ask me about this type of stuff I tell them the same thing you mentioned in the video, choose the technologies you need to accomplish your goals. I'm a true believer that a storage solution should be a storage solution and a compute solution should be a compute solution. Mix-n-match to fit your use case.

[@montecorbit8280]

You tested TrueNAS's VM and found it lacking. I have been told that ProxMox can do NAS. Do you plan on doing a video for that, or have you already done that recently? It seems like it would be a good compliment to this video.... Thank you for the video, and thank you for your time.

[@bw3388]

Thanks one big aspect this review seemed to gloss over. You can say TrueNAS Scale's VM features suck, but Proxmox as a NAS OS sucks even more... unless you virtualize TrueNAS under Proxmox... which is then going against a number of experts that warn not to do that for long term production use... including TrueNAS themselves. If cost isn't a consideration, then seperate and isolate your NAS from your VM infra and have the best of both worlds. But most of us homelabbers need an all-in-one... and that means compromise in one direction or the other.

[@maurvir3197]

@@bw3388 I'm still learning in my homelab, so this isn't a snarky/rhetorical question, but why? Windows servers are generally always run in VMs now and perform fine when storage controllers are passed through to them. Why does TrueNAS not recommend going the same route?

[@eDoc2020]

@@bw3388 From what I can tell ixsystems says it's fine to run TrueNAS in a VM, as long as you pass through the entire storage controller.

[@bw3388]

@@maurvir3197 I'm not an expert, but I believe many have seen unstable vt-d (pci-x pass thru) on some motherboards, SMART data not making it to TrueNAS, something going wrong and having to deal with the VM aspects add more headaches, etc. It's also a bit hard to argue when Scale has basic virtualization and containers as part of it's base OS, so do you really need the complexity of adding in xcp-ng or proxmox as a base layer? Again, I don't claim to be an expert. But running Scale on bare iron with a few linux VMs and a few containers is working just fine for me for a few months now.

[@Darkk6969]

It's one of the reasons why I run TrueNAS on a dedicated hardware strictly for storage as it was made for in the first place. I am running Core and will soon move over to Scale. I use ProxMox for everything else.

[@UnwalledGarden]

Everyone has their own use cases and budgets to manage, but I lean on the Unix philosophy when it comes to NAS - let it do that one thing well.

[@sam_spaxter]

Right off the bat my number one hangup with hosting VMs on TrueNAS Scale at home is that it limits the hardware I can use. I frequently build these homelab systems from older parts, and TrueNAS doesn't let me run older drivers when needed for compatibility. At least not without a lot of extra effort. Love TrueNAS Scale for many things, but not as a primary VM host.

[@McTwistedTwisties]

I run truenas on bare metal and run containers instead of using it as a hypervisor. I have a beefier server running as a hypervisor instead, however, I can't wait for the switch from k8's to docker later this year, and getting access to containers and apps like Authentik and a better implementation of Tailscale. Right now, I'm just running NextCloud and my NGINX proxy manager with TrueNas and it seems to work really well.

[@LtGen.Failure]

i'm running TrueNAS scale as a one stop solution with two VMs. One for jellyfin and plex with an a310 passed through. the other one for docker and running a printserver with an usb pcie card passed through. afair the setup was a bit tricky and i ended up doing the passthrough config manual but aside from that all is running perfectly stable. i had the same problems using spice but it worked well eough to set up ssh.

[@jc57965]

Thanks for this video was debating between Truenas and Unraid for a new NAS build. I don't like the chunkiness of virtualizing truenas in proxmox and found that unraid may not give me the fastest read write speeds but does everything well enough in my lab in an easy setup without sacrificing a drive bay for an os.

[@dominicheynderickx722]

Once ran a VM (simple CentOS installation) to serve as a Plex front end -- No hardware decoding. It worked without a fuzz, but it hadn't that much work to do. I set up the VM with auto patching and left it running. The only thing special I had going on was a an in-VM mount of my multimedia datapool.

[@adam03957]

Having had truenas apps drop out on me while migrating and boot issues on updates to scale in the past I’m 100% back to running truenas inside proxmox for remote management still being available in maintenance

[@chaosfenix]

Thank you for calling out these issues. I actually do use Truenas Scale for virtualization but I have definitely found it as buggy. I will say that I think it is a bug that your password is being shown. It is definitely hidden when I do it. The Spice console is definitely infuriating though.

[@CSIClough]

I have used truenas for hosting a couple of headless Linux VMs when doing some full environment dev but didn't want it on my other machines. And I totally agree, it's a flawed implementation of a hypervisor that's trying to be vmware esxi but doesn't know how to.

[@davethetaswegian]

OpenMediaVault with the Proxmox kernel, Docker and Compose work well for me as a NAS first, virtualisation second setup. Although, my virtualisation needs are modest.

[@Boburto]

Also what I run. I even installed cockpit for better control of VMs. OpenMediaVault is underrated.

[@davethetaswegian]

@@Boburto Agreed, OMV has been doing the Debian based NAS for a lot longer than iX Systems. I have been using it for more than a decade now.

[@adamcadd]

The apps are hot grabage too. Upgrades end up locking up and failing to restart if you look at them sideways. Have to reboot server. I'll stick to my xcp-ng for virtualisation

[@THEMithrandir09]

Honestly, I'll never run anything except purely ephemeral workloads on trueNAS ever since an update wiped everyones data. Sure, we had backups, but this kind of breach of trust takes over 10 years to earn back in my book.

[@PoolOfTrees]

Thanks for the very straightforward and honest review Jeff. It's good to know that there are some areas TrueNAS shines in, but others where you're probably better off with some other solution. On the 'British beers' front (Scots and Orcadians are probably going to hate me for lumping them in that category), but have you tried any from the Orkney Brewery? I've enjoyed Puffin and Dark Islands, and just found out that their website tells you the hops and malts used, as well as where it sits on the bitter to sweetness scale.

[@wolf3five]

I have had no problem doing vm on my truenas scale just the spice, but I did a remote viewer to the vm and no problems

[@CalConrad]

Thank you. Please do more content on Proxmox

[@eric-seastrand]

I would love a similar one about the new Docker support in Scale. Like : can TrueNas replace Debian as a docker host now? Always wanted to bring my containers closer to the storage.

[@hakovatube]

Thank you, and please always stay objective.

[@JoshDike-lx8gl]

Checked mine and the vnc password is definitely obfuscated as you type it in 24.04.2. and the character limit is not 8. What version are you using in this demo? I will admit using this as a hypervisor is not the greatest experience but I only have one or two machines that I need to run as VMs instead of containers and for me it OK enough.

[@MWazei]

I use a Delock M.2 to PCIe Adapter with an LSI HBA plugged into it (use pcie riser cable if needed)

[@ShiggitayMediaProductions]

I run TrueNAS Scale (here on out I'll refer to it as TNS ) for my NAS and Plex (and related tools) needs and it shines with that... In the past I have virtualized OPNsense as my router solution using TNS' KVM/VM stack and while it worked, it was very VERY unstable. If the VM was to go down, I had to reboot the entire TNS system (thus also taking my entire network down with it) to get it running again for a few hours , and rinse and repeat. I agree with you Jeff. TNS is great as a NAS/container OS... It's not (currently) so great for mission critical VMs in any industry or use case. I guess if you need that occasional Windows VM handy for a certain task it'll work, but even then, as you've described in this video, it's better to just use something like VirtualBox or actually boot into a native Windows (10/11) install whereever possible. I'd love to have one system to run a VM for my arcade cabinet as well as be my Plex host.. I know Proxmox likely can, but that's a can of worms I won't get into here (Proxmox, and my arcade cab's Windows 10 install itself). With that said I also agree with you Jeff on "I'm not an guy". I don't work in IT... persay... I work for MicroCenter so I guess I'm IT industry adjacent? I grew up with all the emerging tech around me and I use whatever it takes to get whatever task I need to get done, done. I do PREFER macOS/Linux/Unix-like OSes over Windows, but if the macOS/Unix-likes can't do what I need to do, Windows is my only option, and I'm fine with that. Do I run Windows 11 as my primary OS? No. Not at all. Even when I wanna hop onto Twitch and stream a game I use macOS via my M1 Mac mini because OBS and all the related rools have come a long way in the past few years. But yeah... I'd stay away from TNS' KVM/VM stack until they improve all the stuff that Jeff has mentioned in this video... Once it's all perfected I MIGHT try and run everything using it? We'll see. Good video Mr Jeff.

[@Cpgeekorg]

you can get m.2 4x pcie risers - you put a normal lsi 9400-8i or whatever into a physical pcie slot on the case (that has no slot), you run the riser between the card edge connector to the m.2 slot.

[@majoryoshi]

The issue is that, if I remember correctly, the HL-8 only supports mini itx motherboards and consequently only has one PCIe slot, so even if you were to use a riser you still need a place to mount it

[@Karatektus]

im using scale mostly for apps works pretty well for me. I only have one actual VM though with 1 measly gig of ram and its running home assistant. had no issues. but also no real demands.

[@project_mini_hero]

I can agree, TrueNAS as seems a bit lacking for a full hypervisor. I run XCP-NG but host a few vms on TrueNAS to run some automations against my actually hypervisor but I am anxiously awaiting the docker update for TrueNAS to retire those vms.

[@likwidflame]

TrueNAS for storage, hosting Proxmox as a VM, hosting all services on SSD with pinned CPU threads in TrueNAS. Seems to be fine, but I know I could use some different hardware to push the limits.

[@likwidflame]

To add detail: i5-10400 64GB DDR4 3200 ZFS On TrueNAS: 2 WD Gold 10TB (Deep long term) Mirror 2 WD Blue 8 TB (Not as important) Mirror 1 Barrracuda 6 TB (Downloads and ISO's) Span 1 Samsung EVO 1 TB Span Running: Proxmox as VM Jellyfin as App Damselfly as App PiHole as 2nd DNS App Proxmox is running on a few pinned CPU threads, allocated 32gbs of memory, CPU Passthrough. It is serving: PiHole as Primary DNS Wireguard Mumble Jitsi Subsonic I have some issues with stuttering video which isn't easy to track back. Maybe this is related to how fast the data is retrieved, relayed. However, this doesn't happen all the time. Otherwise, the system is stable and serves NFS/SMB at NIC Speed.

[@csopelario]

Watched this today (on Taco Tuesday) and now I know what i'm getting for lunch...

[@testdasi]

Given the way THIS server is configured, it isn't really an option to run Proxmox and then run TrueNAS as a VM (which would require PCIe passing through the HBA for disk health monitoring - which is a critical requirement for a NAS, not ECC memory). One critical thing to note is that TrueNAS runs ZFS and even in 2024, there is still a bug with ZFS that prevents proper isolcpu functionalities (i.e. reserving some CPU cores for VM and preventing ZFS from using those cores doesn't really work). So running a VM on TrueNAS means you WILL run into high latency during high IO because ZFS will use whatever cores it wants, at any time it wants. This happens EVEN WHEN your VM exclusively uses passed-through NVMe drives. Running TrueNAS VM under Proxmox will mitigate this bug, at least to some extent. Under high IO, your isolated VM will 100% still be isolated from TrueNAS. There will still be some latency if your Proxmox storage is also on ZFS but presumably that would be on a fast NVMe (or at least a SATA SSD) so it will be order of magnitude less than when your VM having to wait for TrueNAS ZFS having to wait for a HDD to respond.

[@Karakofirespartan]

@level1tech I summon thee Edit: I just grabbed the BD790i SE can't wait to build with it. The SE slightly cut down version of the regular board.

[@Marc.Google]

Jeff, highly recommend checking out Sleeping Giant Brewery and Lake of the Woods Brewery, both based out of Ontario, Canada for some excellent craft brews! 🍻

[@bikerchrisukk]

Thanks for this Jeff, I played with VM's on Scale (no experience with any other hypervisors) and found it laggy and buggy, thought it was my hardware or my lack of competence. I'm wondering if the Debian+KVM is the easiest to use or if Proxmox is a better bet? For storage I've been very comfortable on Core for about 4 years now, while I do have a VM with that, it's only for TailScale. VM's are a curiosity, not so much a necessity! 🙂

[@gordan79]

Are you sure your 8+ character passwords are not getting silently truncated to 8 for vnc?

[@hotstovejer]

First option: Proxmox Second option: KVM on Ubuntu, being managed via Cockpit Third option: Truenas Scale

[@yukanolive4727]

thx for the video ! for me VMs only Proxmox , and File Sharing etc with TrueNAS. i had many problems with vms and apps in TrueNAS Scale , and on the latest version many more.

[@Yessurrr]

For an AiO solution try Unraid. Great Hypervisor, Full Docker Support, ZFS Pools or for power saving purposes the good old unraid Array.

[@omnipotech1557]

What great timing on this video, as i am considering between Proxmox, TrueNAS and Unraid. I think my ultimate setup will end being Proxmox as the hypervisor, and a VM of truenas for the storage.

[@kenm3ng121]

this eat up alot of the cpu ram and bandwidth， better keep nas and proxmox in separate machine， i run unraid as vm inside proxmox as a backup and photo album，the speed is SLOW compare with bare Metal truenas

[@karserasl]

Tried that. Truenas as vm is weird. It worked, and kept working for a some time, then when the ram assigned to the vm got almost full, because that what truenas and zfs does, it just locked the vm. And this wasnt just once. It happened alot. And sorry but, everything was using the shares from that vm. You cant have your storage suddenly become unavailable. Not worth it. Either split the storage to other machine, or do something else. (Im doing btfs on proxmox host now and bind mounts)

[@krz909_exe]

Allmost the same config as mine... I have passedtrough a PCI Sata controller to a W11 VM and the speed picked up.

[@JordanHinks]

I currently use Truenas CORE, and soon looking to switch to Proxmox - Truenas for me was easy and a first step onto the ladder but I experienced odd issues with VNC (noVNC) for VMs where it wouldn't always work or pass through the correct key commands according to my keyboard layout. Other strange issues were that with statistics graphs for quite some time only worked in Firefox but that was eventually resolved.

[@antonterekhov9389]

I use both setups: 1. Main server: Proxmox is the hypervisor, and TrueNAS is a VM with passed-through storage. I have ~10 VMs in Proxmox. 2. Secondary server for backups only: TrueNAS is the primary OS, and only one VM - Proxmox Backup Server. Why? TrueNAS is easy to recover/migrate to a new server, so I always can restore it w/the Proxmox Backup Server. From here, I can restore my Primary server.

[@6LordMortus9]

I currently have an old Dell R710 as a TrueNAS server. My "main" system that I was using for ProxMox is currently not running as I plan on transplanting the innards to a 1U rack. I plan on using an iSCSI disk on the Dell to store VM's in ProxMox. I would also like to know how to get some of the beer you drink! :)

[@cracklingice]

It is stated that there is a cap on ZFS cache space with RAM in virtualized TrueNAS but the limit is not stated. As I have been dabbling with Proxmox on an i3-7100 dual core with 16GB of memory (yes it is brutal) - while my NAS is a 16 core X99 with 256GB of registered ECC, completely overkill for just a home NAS. Oh and for giggles - it's got a 25gig connectx-4 as does my desktop. I cannot saturate the 25gig link but I definitely can go past 10gig. That is as long as the file is in the cache already. Something I'd also like to understand more about is iSCSI and how it interacts with the ZFS cache. Does the entire iSCSI volume need to fit in the ZFS cache or can it reference parts of it, ect.

[@turbo5546]

I ran proxmox inside of truenas scale and ran that for months without any hiccups, but I was just running some network management vms. But I did have this as part of a proxmox cluster.

[@nadtz]

As much as I love TrueNAS this is pretty much the conclusion I've come to as far as using it as a VM host goes. I run TrueNAS Core on bare metal on one box and Proxmox on another and have been pretty happy with that setup for some time now. That said I'm pretty sure VLAN's do work and the new version of TrueNAS Scale should be coming soon and should address some of the issues you saw.

[@bufanda]

In regards to the VNC Password, when setup during the wizard with a 16 character password, did you try to just use the first 8 characters later on to access the VNC session. I wonder if the wizard just omits the last 8 characters and the Password field also only uses the first 8 characters. I have seen this happen on other products that act like you mentioned it.

[@cras2298]

Great update. How would you have configured if you did not need the additional Intel GPU card?

[@grimace66]

@CraftComputing I'm curious as to which version did you use?

[@pvalpha]

I guess I look at truenas to do one thing for me: host storage and the services directly related to storage. It isn't what I'd look for to be my virtual host. Would I like that? Sure, I could have a couple use cases for that. But I wouldn't *trust* it for that even without the problems you encountered. I trust it for storage.

[@christians3441]

I've tried it before but ended up with a small Lenovo PC as a dedicated virtualization host and a Ugreen NAS as a TrueNAS Scale NAS. The only service I run on TrueNAS now other than the file services themselves is Plex out of laziness and for the direct drive access. I would consider TrueNAS with VMs for small "deployments" at for example a family members place where the primary need is storage and where containers and VMs are just an added bonus on top.

[@MD2389]

I'm glad I'm not the only one that has had issues with running a VM on Truenas. I tried running a VM so I could run a stripped down version of windows, just so I could run a game server not on my main rig. I honestly found the UI for setting it up incredibly clunky and just gave up after fighting with it for over an hour. It flat out refused to read from the ISO, even thought it could read it's location, and for the life of me I couldn't get the GPU to pass through. To this day I also have to go in and manually update running applications, even though they are set to auto-update on their own. Don't get me wrong, I really like Truenas as a whole, but it seems that anything beyond basic functionality just feels half-baked. I will still keep running it bare metal for the future, but I'm definitely going to use something else whenever I need to run a VM.

[@JustinLeif]

Hey Jeff, On the Minisforum BDI790i... I'm assuming you can get to the m.2 wifi card and remove it? I thought that if we wanted to virtualize, say a OPNSense router... we could swap the wi-fi card out for a second 2.5gig nic, say a m.2 2230 2.5gig A+E key and then just install the nic port in one of the case slots. Thoughts?

[@bigshotking]

I've noticed that with the VNC window that if it is left alone with no input for 3-5min (even with the window still active) it will become unresponsive to your inputs. It seems to be some sort of timeout when there is no input from the user.

[@Big7Cheese]

i experienced all of the issues he mentions. I knew i was in for a rough time when Spice K&M was barely functional, but the pcie gpu passthrough that just ended up not working properly did it for me. Just looking to have my NAS as its own separate box at this point

[@chromerims]

Thank you, incredibly helpful 👍 Kindest regards, neighbours and friends.

[@chromerims]

9:07 -- nice reminder to not over-provisioning memory to a VM.

[@MatthewHill]

I run TrueNAS as the "bare metal" OS and I've used it for a couple of light virtualization tasks--and it's always worked perfectly fine for me, albeit a bit limited. BUT, this was the older "Bhyve" virtualization stack. I've never tried the newer KVM-based stack.

[@rthomasv3]

Tried TrueNAS as a VM host about a year ago and it went pretty bad. It was a great NAS, but yeah VMs were really unstable. I decided to just use two machines - one with Proxmox for VMs and a separate TrueNAS machine. It's been great so far, and I would definitely recommend that over virtualizing TrueNAS if you can. I also use the NAS for VM backups with rsync as a CRON job for offsite copies.

[@holdenluci]

You mentioned everything but UnRaid, it is not free but it is my homelab all-in-one solution! Granted, I do NOT have any VM running in it, but I do have dozens of containers. Have you tried unRaid?

[@Morten.Bremvåg]

Been running scale for the last couple of years and been running VM’s for school work the last year. It’s been fairly good, even setting up hyper-v on it. Spice is terrible thought, so I use ssh and Remote Desktop and connect to the vms that way. Also annoying that you have to create a network bridge for vms to be able to see truenas network shares on the network. I mean the vms can see your other computers just fine, just not truenas itself. But it was more stable dedicating an own nic the vm’s instead of running it on the main nic of truenas (heavy torrenting could make you lose connection). Overall I am happy with it. I mainly run truenas for Plex and torrents. Now I can also do my studies on it and host vms, so I can do school work from anywhere through the vpn I setup for my home network.

[@teagancollyer]

Hardware Haven did a video recently on updating his HP SFF server with modernish hardware. In that video he too was using m.2 to sas adapter cards but his had the ports lying horizontal on the card itself and not vertical like the card you tried

[@torak456]

I feel perhaps silly asking this, why do you not use the integrated graphics on the CPU to do transcoding? Is it because you can’t pass that to a VM like you can with a dGPU? Is it because of software support, or performance?

[@joseph3164]

I run truenas scale bare metal, and tried for months to run a VM for home assistant. It was a terrible/buggy and unresponsive experience and I ended up just putting home assistant on a raspberry pi instead.

[@1leggeddog]

Your "telling it like it is" about software solutions is why I'm subbed to you. Its easy to have a bias in the tech space. Its hard to recognize it and preach caution when you don't always know about the end user's needs and utilisation.

[@postnick]

VM's in Truenas is like pulling teeth! I want to use it but god its rough. And I've tried it just with server a few times but then networking gets complicated, I just find it easier to keep truenas as storage and proxmox as VM. You know a good VM one could use - Setup a Proxmox VM as a third vote in your system if you can't have 3 full systems.

[@asbestinuS]

I would be interested to see if when you use Proxmox as hypervisor and do the truenas-VM thing, is the performance of the VM the same as in your test?

[@truckerallikatuk]

I've run VMs on Truenas Scale, they were a pain to set up in earlier versions, but its been getting slowly better. I never tried to run anything that needed passthrough. I ran Proxmox Backup server on it for a long time as it felt better to me to have the backup VM running on a storage machine.

[@shabadabadoo4326]

I too found TN's virtualization problematic, so went with proxmox. You talked about virtualizing TrueNas in a VM under ProxMox. how well does it handle serving a datastore from TN to a separate VM running docker containers?

[@jttech44]

This turned out wonderfully, though, part of me was hoping for more hackery, like a PCI switch or something. I suppose actually having it work and be reliable is... nice, so, 45D's solution is no doubt better. That said you could probably find an M.2 card that'd support the SFF cables well enough, likely with right angle cables. Like you said though, more than enough for the spinners you'll be using. If you wanted all flash... well, there are likely better options there anyway.

[@MortenEghj]

Thanks for an good video. One video I will suggest it is an simpel video for how to setup your network IP space and how to imbed Proxmox. The IP dialog and IP setup in Proxmox is still confusing.

[@docfro9256]

Have your tried Unraid before? It's pretty basic. I think it's good with VM and something I'd recommend to new to doing home labs