My 32-Core 64-Thread Home Server MISTAKE

[@CP-gy8ms]

I honestly want to see someone take a build like this and push as many dedicated game servers on it as possible until performance is effected.

[@planetjupiter2551]

Linus did that

[@stinkybad8261]

@@planetjupiter2551 whats the name of the video

[@w花b]

​@@stinkybad8261 We'll never know

[@kienanvella]

The motherboard/bios weirdness is why I ultimately landed on a Supermicro X10DRi when I built my system (dual E5-2680 v4). well, that and the pcie slot layout + pcie bifurcation (oh yes, lots of goodies in this system). Proxmox host, Fedora guest w/ pcie passthrough for GPU and dedicated USB card. It works really, really well as a combo workstation/server.

[@Steve25g]

yea, also have some supermicro boards around, supporting exacly what he's looking for. Also have some gen7 HP and older IBM, and they all have iommu , even while running an older x5xxx cpu's

[@MarceloAlcantaraX]

How much power does it use?

[@owen75]

I currently have the same setup. About to attempt to use it as an all in one system as he mentioned towards the end of the video. How is your experience so far?

[@owen75]

@@MarceloAlcantaraX I have a similar setup and mine idles at around 80w running truenas core with 6 NAS drives attached.

[@thecraggrat]

Just ordered similar parts to make up a server to pay with when I stumbled across this - immediately thought WTF have I done! After calming down a bit I looked at how much I'd spent on parts specifically for the server and those I couldn't repurpose for other systems down the line, or use in other existing stuff as an upgrade and I'm out ~$4-500. I do intend to make it my backup NAS so that saves me a similar amount to what I spent on it anyway, so I reckon it'll be a wash. Even with additional power demand due to server parts reckon I can get the idle power down to ~25-40 watts at ~$0.14/kWhr, so ~$0.14/day at idle and $0.50/day at load, even at 3x that I reckon that's less than the TV consumption... so the power use isn't worrying me. In exchange I get something to play with and that is useful too, and I'd likely be able to get my money back if I wanted to sell the parts too. (Estimated off another Xeon system running an E5-2697v4, which runs cinebench 24 @836 for the multi thread test score) If you are still not put off by the video, the "cheap bits" I got was a Supermicro X10DRi - T4+ that came with 2 CPU's and heatsinks for $178! The reason I got this board is it has 4 10Gb ethernet ports on it already and I want to move in this direction for my internal network, this alone is worth ~$130 in future spend reduction... To help if you are doing this in future, the MB is 330mm x 345mm or about half way between the SSI-EEB and SSI-MEB sizing dimensions. As other people have said this is a rather hard fit to find for a chassis, which quite often don't give you ALL actual dimensions they'll specifically fit. They also are usually expensive, so I was really happy to find the Phanteks Enthoo Pro 2 Server Edition specifically fits this size of MB at a reasonable $160 on amazon and has plenty of disk space for the NAS too. I'll try and update with some actual power readings...

[@adwadwadw758]

I like that every video of yours is around the things I do, you actually explain things good. and I also got to build that kind of "server" and got an RTX for pci passthrough, and wanted to die when I found out that it doesn't have IOMMU

[@ntgm20]

Xeons are neat processors, but by the time us mere mortals can afford them (used), we'd be better off buying more modern mainstream consumer CPUs. I'm speaking from experience, from just a single CPU X5650 system, but it was a power hog, without the returns I expected. Thanks for sharing your story.

[@R055LE.1]

I can second, third and fourth this. I have 3x Dell r610's and an r510. There are some DevOps tasks that work better on a single i7 4770 CPU. Let that sink in. It may not have the same core count, but a modern Ryzen system blows those old enterprise cast offs out of the water. They're fine for getting started (though I certainly wouldn't recommend picking up 4) just to get a feel for how the enterprise was, same reason I picked up Cisco c3750G switches. But for performance get something at least 7 years further into the future. For reference I replaced all the CPU's with Xeon L5630's which launched in Q1'2010. That means Q1'2017 is the oldest I could recommend for any real world real work tasks.

[@gastonhitw720]

amd cpu's might have some problems with vm or in my case network virtualization, i only want intel cpu, but why would a xeon 2699v3 be worse than a modern cpu, obviously the embedded technologies are better for modern cpu's but when it comes to cheapness and cpu cores isn't that better? i don't really know if the low cpu speed could really lower a vm performance, haven't tested that @@R055LE.1

[@deyeatdapoopoo7582]

Not really. The problem is that you're basing this on an experience that while true is not nearly as relevant anymore because improvements aren't nearly as big as they used to be. During that era from 2008 to 2015-ish each new generation of CPU brought big increases in performance, power efficiency and value for money since prices weren't going up (adjusted for inflation), so you were better off buying new, or something used that was only 1-2 generations old. That's not really the case anymore. Prices have skyrocketed and while performance and efficiency have kept going up it is not anything like the pace at which it was happening from the 00s to 2015 where after that it started to stagnate. Using your example, going from a 1st to 2nd gen Xeon meant a massive upgrade in performance and power efficiency. Since we're talking servers the best metric to compare is performance per watt, and PPW would've been a massive jump of around 40% just going from one gen to the next. If you look at more recent architectures, going from a 5th gen to 6th gen Xeon only yields about 10% higher PPW. On top of that, Intel massively restricted what chipsets would support which processors starting with 6th gen which limited options which in turn lessened competition. Because of that, it's much more expensive to get a Skylake Xeon system vs Broadwell despite them only being released 1 year apart and the relatively minor performance and efficiency improvements. For regular tasks or gaming, getting newer cheap processors makes sense but if you need cheap multi-threaded performance the 4th and 5th gen Xeons make sense. You can buy a used workstation or build your own for cheaper than you can get a 12th gen Alder Lake i3 platform and they will trounce it in MT performance. A Xeon E5-2682 v4 can be had for under $25 on AliExpress and is a good example of that. They are absolutely not the end-all be-all, but they have their scenarios where they make sense. It just doesn't make sense to get the 1st to 3rd gen Xeons anymore, but the Haswell and especially Broadwell can make sense.

[@bruceallen6492]

BRAVO! Well done! Everyone has issues and it's great to see how people correct for them.

[@jonjohnson2844]

Electricity is so expensive in the UK now that I can't justify having such a beefy CPU idling most of the time. I actually downgraded my unraid server from a 4790K to a 4570T - the reduction in power consumption is dramatic but, performance-wise, it's not that noticeable...I feel maybe Plex streams take a few seconds longer to buffer the initial transcode but I've not done a direct comparison.

[@Jimmy_Jones]

Yep. My tariff is equivalent to $0.3662/kWh

[@RaidOwl]

I love my 4570T lol

[@jonjohnson2844]

Yes a very underrated CPU and it only cost me £20 with a warranty from CEX. I have about 25/30 docker apps on it for a number of purposes and it handles everything fine. I actually tried the equivalent Xeon CPU first but my mobo won't boot without a GPU and I thought maybe even having something like a GT710 in would defeat the object of reducing power consumption as I presume it still sips a few watts even though it's not being used.

[@jonjohnson2844]

@@Jimmy_Jones Yes my tariff went from 13p in January to 28p now and looks like it will go up again in October, it's crazy.

[@blazingmatty123]

I'm not sure if this board specifically fits the form factor since the chinesium ones don't always fit form factors specifically, but usually these kinda nearly dual wide (12x13") boards are generally a form factor of SSI-EEB

[@aknightstale2248]

Corsair's Obsidian 900D fits SSI...great used bargin case

[@gustersongusterson4120]

I'm glad folks have been making videos about all these Chinese X99 motherboards. Not that you're in the market... but you should check out the dell workstations like the T7810. I picked one up for less than the price of that motherboard. It has tons of pcie, has VT-d for IOMMU support and is actually reasonably compact compared to that chassis. It only has room for 3 drives and the PSU isn't designed for power hungry GPUs, but I solved that with and external PSU with cables run through where the optical drive used to be. I need to find a solution for an external JBOD enclosure that isn't an enormous rack and I'll be good to go.

[@kamranguliyev4258]

You can flash custom bios on this motherboard with undervolting+unlock turbo boost on all cores+ram timing. A year ago I builder for my friend 2x2678v3 with 8-channel 64gb ddr4 ram and used ch341a programmer to flash bios. -70mv undervoltage. Works great

[@kamranguliyev4258]

Though if building today I would choose 2696v3. Xeons have limited frequency by their tdp. So without undervolting even if you unlock turbo boost you won't get max frequency under full load. Also depends on your use cases you can turn off cores for more Hz as well. For 2696v3 if you turn off cores to 10c/20t you can get 3.8ghz on all cores which is the max for this cpu. On 12c/24t it's usually 3.7ghz but if you are lucky with the binding you get 3.8ghz. 14c/28t - 3.6ghz. Depends on your tasks. Turning off cores can be done in OS, so you can use all cores in programs and 12/24 in games.

[@KevinWhiteJr]

I'm building the same type of system (same MB, E5-2678 v3 and eventually 128GB RAM). I will be using this as a virtualization server to run different environments (job stuff). It is good to know that the power draw will be a thing, so I will be watching that. And I can second the frustration of looking for XL-ATX cases. Awesome video!

[@lllongreen]

Look for something newer ! With thr electricity prices for the next couple years it's not time to be cheap on the CAPEX you have to look at the OPEX !

[@junaidtanoli1]

what is your power draw during boot time and idle run.

[@KevinWhiteJr]

@@junaidtanoli1 I haven't really checked. I need to get a gauge for that, but it has only been up for a few weeks. I think I'll take a look at my power consumption in a month or so.

[@rdsii64]

I know this is a really old video. With that said, its been my experience that these Ally express motherboards that repurpose server chipsets are hit and miss with virtualization. Sometimes you get boards that won't support passthrough even though the chipset, bios, and CPU's "officially" support it.

[@brentpolk8331]

I have 2 of these built for render farms and with 20+ core dual xeons, it's a formattable beast!

[@jonathanmayor3942]

I'm building, just like you a server with pass-through in mind , and I use old Skylake-X and threadripper 2nd gen to get the cost a bit down and here what I found out : - Gigabyte motherboard; you can set the primary GPU pcie line like if you plug the quadro in slot 4 and other 1080 ti on all the other you can set primary on the quadro - Asus WS : known to have the most lines, sage 10g FTW

[@rdsii64]

I like building cool stuff on the cheap. A 32 core machine just to mess with stuff is just what the doctor ordered. I have a spare rack chassis that will fit this board, so who knows what I might do this time around.

[@schemage2210]

I could see it being an insanely awesome blender rendering rig, if you are reliant on cpu rendering that is, after all not all of us have access to high end 3090's or the like that usually make rendering quicker.

[@IndellableHatesHandles]

Compositing is still entirely CPU-based, and it's very intensive on certain projects.

[@QuarKSonTV]

Maybe you could make a video about building energy efficient NAS? Raspberry Pi is great, but it's not really powerful

[@RaidOwl]

🤔🤔🤔

[@bopal93]

That would be great

[@Doesntcompute2k]

@@RaidOwl Those higher-core, lower-end Ryzen have become the best for a lower-power (not low-power) NAS. The issue with a lot of the SBC is ports: they don't have any USB 3.1Gen2 or (any/many) SATA. A decent ITX or so AMD mobo with an 8 core Ryzen has enough SATA3 ports AND has a PCIe x8/16 you can place a LSI SAS2 board flashed in IT mode to handle 4, 8, or 16 more SATA drives (including HDD, SDD). No one I have found YET on KZbin has truly done a "build this from scratch" video in a configuration like this; It would really help a lot of people. Using the newer Ryzens, with common sockets, you could easily get a mobo supporting from 6 to 16 core CPU allowing people who want to use VM/Dockers on their NAS all the horsepower they need. While the energy use is still as low as possible. And these mobo will also support multiple 1/2.5/5/10Gbps NICs. Just a random thought. :)

[@ericmartens8151]

I wouldn’t entirely write off the chinese x99 boards. I have been running a 2670v3 on a machinist board for a year now and haven’t had much for issues. IOMMU works with no issue by enabling VT-d as you have shown.

[@jaynj908]

I beleive the Eatx standard has never been really etched in stone

[@Cmnore]

As for the Abomination-sized case? I feel your pain, brother. I was lucky enough to stumble across the TT Core X71 at a local computer store that was closing its doors for $150 I has begun building a machine around an HPATX sized case which is 10 PCI slots tall and even wider than EATX. I scrapped the z800 board and grabbed a Supermicro X8-based EATX board and it fits salaciously. I'm hoping to consolidate MY pile of smaller PC's into one machine. The case dilemna may just be irrelevant should I choose to go 2U down-the-road. The chase continues!

[@JohnSmith-ty4yt]

If you can, try getting your hands on a used Supermicro X10DRi. It supports SR-IOV which should solve your IOMMU issue. They're more expensive than the chinese boards but are still worth it since those are some of the boards that are being harvested for their chipsets. It will also be EATX, so keep that in mind. There is also a Ukrainian KZbinr called Miyconst that has more in-depth knowledge about these boards, try sending him a message and he might be able tp help you with your IOMMU issue.

[@SpookyLurker]

I guess I made the smart move by watching said video from Jeff. It struck me as one of those sketchy builds - so, it wasn't even a thought.

[@nyccontrabass3489]

Just got a dell T620 and I’m having a blast playing with it. Payed $400 total. 256 Gb ram and it came brand new with 6 brand new dell drives 500gb each.

[@Frog3136]

Interesting to see how much the idle power draw was for that which is ridiculous. I'd say gut the system and see the board/CPU/memory to fund something else. With that kind of power consumption it's never going to be a friendly system

[@RaidOwl]

Agreed 🙁

[@Jimmy_Jones]

My R710 draws about 180w idle. About 240w all out I think. I only turn it on when I need it. Otherwise it's Pi's.

[@Kajukota]

Hey Raid Owl, not sure if you've ever checked out his videos, but take a look at Miyconst Hardware's videos on these chinese motherboards. He tests a lot of the features on them, including comparing pcie lanes, and some server features. He's a Ukrainian creator, and has recently dedicated his earnings to the support the defense effort in his home country.

[@RaidOwl]

Yeah I watched his stuff when I was troubleshooting. That guy is awesome and has a lot of good stuff. He puts so much work into these boards and his content.

[@Doesntcompute2k]

That's the GUY! Yes, he does a great job! I was trying to think of his channel name. Thank you!

[@ozymatic3291]

Love the tshirt & the build!

[@Thewickedjon]

thanks to channels like this one and craft computing, i've been slowly building my unraid box, on a gigabyte x99 board, with an e5 2690v3 and 32gb ram, x6 3tb hdd's, on a very very old corsair case i got off FB marketplace for 20 bucks, .......... and I hate you guys... but I feel im close to getting it almost set... so I keep watching

[@aLlbLck3]

Random viewer here... So after a year, how is your server doing ?

[@Thewickedjon]

@@aLlbLck3 ???? Well the old gigabyte x99 board died, so I now have a chinese x99 mAtx board (machinist) > the case is now an 8bay hotswap nas case from amazon, (which I love) > 128gb ecc ddr4 ram, dual 4tb pci-e nvme's in a pool for unraid cache , and still x8 3tb wd red drives for raw storage. still same e5 2690v3, oh also have a quadro p600 for transcoding. its working fine, i guess, it does what it's supposed to do. Hosting a few game servers, personal media , pihole, and a couple vm's, you know what most folk have

[@aLlbLck3]

@@Thewickedjon That's a substantial upgrade my friend, specially the RAM amount. I decided to build my very first Home Server/NAS, with very humble beginnings. Since electricity is kinda high here, I had to pick a low TDP processor. I found a very good deal on i5-10500T, so I went with Intel LGA1200 Socket, since they have intel QuickSync available. - Intel i5-10500T - Asus Tuf Gaming B560M-PLUS (will allow for better 11th processor upgrade down the line) - 2*8GB DDR4 3200MHz Like u, I'm using a very old case 😅, and overall just testing. PS: I also blame these guys for making me want a server.

[@CoreyPL]

I went with a mATX-C612 server board from AliExpress. Single socket 2011-3, Xeon E5-2680v4 (14 core/28 thread), 4x32GB ECC DDR4, 2 NVMe m.2 slots, 10-port on-board SATA controller and 2x2.5Gbit Intel 226-V. I don't even need GPU, since it sports ASPEED2400 for simple VGA (no BMC though). Board supports UEFI and legacy boot, has a connector for TPM unit with support for it in BIOS. Board+CPU shipped was around 150USD. 4x32GB ECC DDR4 RDIMM 2400MT was 130USD locally. Board supports bifurcation, IOMMU (Intel VT-d), SR-IOV. In native Linux it idles around 53W (just 1 NVMe drive installed). For test I've also ran native Windows 10 on it - no problems, just had to install Intel Chipset drivers for C612. So Windows Server should run just fine as well. I'm still testing it before committing to running it 24/7, but so far it looks promising. 14 cores / 28 threads is enough to run TrueNAS Scale (with onboard SATA controller, so no HBA needed) virtualized on Proxmox (SATA controller as passtrough) with some other test VMs and containers - so similar use case to yours, but at much more wallet-friendly settings :)

[@lllongreen]

As other people already said, with the crazy electricity prices that will at least continue throughout this year, its not worth it to get these apparently "cheaper" older CPU'S . There is a reason these old Intel CPUs are sold for cheap on ebay... look at something like an AMD Ryzen 5950x instead, this is a case where you have to look t the OPEX rather than just the initial CAPEX. The 16 Zen 3 cores are quite powerful compared to the 32 old Intel cores

[@Darkk6969]

Yep. It's one of the reasons why I upgraded my 7 year old Intel 4790 to Ryzen 9 5950X as my daily driver Linux workstation. I may pickup another 5950X to run ProxMox but have to locate a motherboard that *can* run ECC memory.

[@trumpsextratesticle8590]

Elections have consequences, your overlords want you to beleive in global warming (so electricity is high) so they can force you into Serfdom.

[@nixxblikka]

You made me laugh twice within the first 90 seconds, some movies dont get me laughing at all - within 90 minutes - also dinner cable joke 😀 Ithink its cool to share this info, since the inner napoleon always goes for: I need 100 gig fiber and optane only storage in my epyc server - then again, we are using it home to play around... thanks for putting things into perspective from various angles.

[@markkoops2611]

Power usage at BIOS is maxed, not idle. Systems assume something is wrong if you're in the bios, so don't engage power management

[@shaunwhiteley3544]

My electric tariff is currently 15p/kWh. An extra 100 watts on a 24/7 system is £131.40 a year. My electric is going up to 36p/kWh. I imagine a lot of people are going to be looking at more efficient new systems rather than old power hogs.

[@RaidOwl]

Agreed

[@trumpsextratesticle8590]

Yes, keep using less and less until you are living like a SERF....LOL Stop voting these Economy crushing "Green new deal" terrorists in office- VOTE THEM OUT! The US hasnt built a new nuclear power plant in over 20 fucking years but the population has added 50 million plus...and people wonder why elecrtic is up and the power grid is failing.

[@marcc5768]

I'm Kew with using used Exons for builds. I actually have a e5-2650 v2 on a Kllisre x79 mobo with 16GB ddr3-1333 (4x4GB) ECC in quad channel. Running Linux Mint 20.3 Cinnamon as my daily driver, no gaming. Built in Oct 2021. As for the beast you built that would scare me on power consumption alone. Tho I have thought about a VM server/NAS build but going for a less power hungry cpu. Maybe an e5-2640 v3 or dual 2640 v3s.

[@84Actionjack]

I went through the same thing and swapped the Chinese board for an Asus LGA 2011 v3/v4 board. I still think a TRX-40 motherboard would be better. Good stuff!

[@RaidOwl]

Yeah a threadripper system would be cool $$$$

[@JasonsLabVideos]

Love it, we both built servers. Except one of my nodes pulls 125watts lol.

[@Radoslav_Stefanov]

Considering what you can do with a cheap second hand Mac Mini M1 with like 5W at idle the electricity cost alone makes these 32 cores a waste. Even an old Intel based Mini is more than capable for most things you are discussing including PCI passthrough. Unless you are exposing your home setup to a lot of users there is really no point. I am using a 2018 Mini with 64gb memory, a 2tb SSD with a couple of USB-C external drives each on a separate bus running all my workloads on k8s and a few VMW VMs. Yet to find a reason to replace it. This thing is an amazing little beast within such a tiny form factor. P.S. - I used to do custom builds myself, but nowadays I just want to do my work without wasting my time.

[@RaidOwl]

The problem with mini PCs is lack of expansion

[@Radoslav_Stefanov]

@@RaidOwl most people dont need the expansion or future upgrade options. If they really do it will be in years when technology has phased out their hardware anyway for more efficient versions. Speaking from personal experience when building systems for others.

[@jj-icejoe6642]

Electricity bill ? 150W at idle and no PCI pass-through ? It's more than my Dell R720xd 2x2650Lv2 P2000 and 10x3.5 HD + NVME on PCI card ! I only have 125W at IDLE !

[@RaidOwl]

Hence the “mistake” haha

[@Jimmy_Jones]

Oof. That power draw

[@gabrielb238]

Dude... I've been looking WAAAY too much at aliexpress and 2698v3, 2690v2, etc... I definitely don't need it (I don't have a need right now for anything I've seen it can be used for) but man have I been wanting to throw money on these parts.

[@choppergirl]

What is IOMMU and what is it important or necessary for running HyperVisor or whatever??? So I looked it up... Overview. An Input-Output Memory Management Unit (IOMMU) is an MMU component that connects a DMA-capable I/O bus to system memory. It maps device-visible virtual addresses to physical addresses, making it useful in virtualization. Okay, so why in the world would a SuperMicro Server motherboard of this scale not have that. I am mystified. It seems like it's only function, to run multiple VM's for Webhosting ISPs to rent out. BTW, there is no such thing as EATX or an EATX standard. There is an ATX standard, and case designers said well you got even more room around it to go further than ATX if you want, so we'll call it space for an Extended ATX motherboard... but there is no such thing as an Extended motherboard standard, hole wise or otherwise. I'd buy it from you for a discount, because I only want to run one OS on it and a Video Editor using all those threads to render on top of it. I already have a 7950x w/ 128gb ram to do that, but l'm curious to see what a 32 core system would do with half the CPU cycles per core. I'm guess it has sort of the same power as my 7950x, just a whole lot more core boundaries in the way. Dunno, probably a waste of money and a dead end, but yeah... that's what I'd use it for... video editing... or at least try it. Probably not a worldclass beater though because the PCI 3.0 slot, if that will even support a high end GPU. I know server mobos are very finicky about what PCIe cards they support.. usually, very few. Resolve swwcks, I use Sony Vegas.

[@Jon-ky6st]

I have the X99-D8 Max, found the prefect case for 40 dollars. Style Selections Steel 3-Tier Utility Shelving Unit from lowes, and 15 dollars worth of polycarbonate underneath it with zip ties. Works great with a bunch of crypto projects.

[@spvvk3621]

Thanks man this video made me back off from the idea. Was going to go either with this or huananzhi board just because I liked the double CPU meme but it's not worth it when I can get a recent medium tier CPU that will blow both of those off the water for my use cases.

[@Jimmy_Jones]

I would only stick with well known brand motherboards. Or at least a known server board. Just something I don't want to cheap out on.

[@BrianMartin2007]

Even a lot of the majority of those Chinese boards with even 8 dimm slots won’t have quad channel memory on a per CPU basis like Iit’ssupposed to have, both CPUs will be stuck at dual channel memory performance, I’m not sure that this is the case with yours, but just something to keep in mind.

[@mamdouh-Tawadros]

It is not a mistake at all. An excellent machine that you turn it on when you need to work, or game.

[@blendpinexus1416]

i'm in a weird spot where i need for a "desktop" machine and a dual socket epyc really seems like the fit. though i could get away with a single socket epyc and then something like a r7 3700 for the other uses.

[@AchwaqKhalid]

I believe a 5900hx mini PC can surpass this setup in many areas ❗

[@RaidOwl]

Lol I bet it would 😅

[@AchwaqKhalid]

#MorefineS500 or #HX90 here I come 💪

[@JasperHasMyBlaster]

case recommendation for other people interested in building something like this, enthoo pro 2 server edition, has 11 pci slots and tones of harddrive space, also lots of fans, like a server case but if you dont want a server rack.

[@cinemaipswich4636]

I bought a combo of CPU 14cores, Supermicro mother board. and 128G or eccRAM. It does more than I need. The new things I bought were Case, 8 HDD's, 2 SSD's for boot, and a NVMe for cache, and a HBA card for TrueNAS. 2x 10gbe's and I lack nothing.

[@Doesntcompute2k]

So, okay, I admit it: I thought this was going to be click-bait because I read the title as "My 32-Core 64-ThreadRIPPER Home Server MISTAKE." Yes, I cannot see well these days! LOL I watched the entire video and took a gander at the title several times again. :) The bit about DaVinci Resolve really made the entire video; Well, that and admitting your "mistake" with the power-hungry 2600s. Man I have around 20 2600s (v2/v3/v4) and 7400s and I have to say that while the core-count rocks for multicore hypervisor/VM/Docker use, getting say 80 VMs on one frame alone in my case, the power hungry processor (up to 145W/socket) and RAM (DDR3ECC or DDR4ECC) is really bad on the UPS and power bill. Especially the Dell 910 with four sockets, 40 cores, and 512GB of RAM. Oh and 8 10Gbps and 2 40Gbps NICs. I made a lot of mistakes, while having fun! So I get it. I'm switching a lot to Proxmox using SFF Ryzen 7 series, and such, and just using more of them. I have six Proxmox servers which don't use the power of ONE Dell r730xd with 24c/48t 384GB. The really nice Aussie over at Tech Yes City ( kzbin.info ) also dived into the Ali-Express X99 mess and he's a good source when he goes hunting for low priced systems. Jeff's KZbin is also. One other person was doing a lot of comparisons of Ali-Express' X99 and such mobo to see if they could get them working--but I forget the channel name. I LOVE how you post in notes the break down of the timeline of your video. If only more people did this....

[@dgtheone]

Enjoy your video’s. Just a correction though - it is pronounced “You-nix”

[@gxtoast2221]

I bought an Epyc 7452 (32 cores, 155w TDP, Rome Zen 2) CPU with 2933Mhz DDR4 RDIMM memory, a Supermicro H12SSL-i motherboard and 4U cooler - all in a Lian Li 216X non-RGB case. I added 2 x 2TB Intel SSDs and 2 x 1TB Kingston SSDs and an ASUS Hyper Gen 4 x16 M.2 NVME card that Bifurcates 4 x4 SSDs. Currently running 4 x 64GB modules with another 2 x 64GB on the way, and another 2 x 64GB after that to fully populate 8 x 64GB RDIMMs at 512GB memory. This thing cost with high quality Seasonic PSU about $2.5K USD but it runs ESXi 8.0U2 flawlessly and provides completely stable IOMMU (AMD's passthrough technology), SR-IOV and Bifurcation. The labs this thing can run are significant and include an entire VMware Cloud Foundation and a host of persistent servers, PNetLabs, Truenas Core and containers for dev and infrastructure services. Totally worth the money.

[@Adam130694]

Huananzhi X99 F8D Plus is like $170 now, supports IOMMU & 2699v3 (18c/36t) costs like $30 now...

[@ufsteropolstero6014]

I wouldn't worry about the 150W consumption in bios. In bios, the CPU runs at full clock speed and voltage which explains the 150W usage for the whole system. In Windows desktop, I expect that to halve if not reduce more.

[@trumpsextratesticle8590]

This, I have a dual socket E5-2699V3 and at near idle, they draw about 14 watts each (28 total)

[@Phoenix_SW20]

No PCI-e pass-through is a bummer. Someone else in the comments mentioned their might be a BIOS mod to fix that but I guess it'd depend on what chipset that Jingyue board is using. There's a guy on KZbin called Miyconst with a massive amount of information for Chinese X79/X99 boards. Some brands use desktop chipsets or have a weird BIOS with missing features. There is a brand called Huananzhi that makes 2011-3 boards with either X99 or C600 chipsets. X99-TF, T8, and F8. They've also got two dual socket boards. One that runs DDR4 and one DDR3. I recently ordered a X99-TF (supports either DDR3 or DDR4) with a 2678v3 to upgrade my ancient Unraid box.

[@Phoenix_SW20]

I did some research and all I could find were turbo boost unlock BIOSes for your board. Odd that IOMMU isn't supported since it should have a C612 chipset.

[@7073shea]

Thanks Owl!

[@declanmcardle]

Men in Black would have been a much shorter film if they said "the Galaxy is on Orion's collar..."

[@ronsafranic5177]

There are a couple of Super Micro boards that would solve your problems. Unfortunately the last time I checked on E-bay the prices were up there. You can get the whole server cheaper usually.

[@JamesT65]

Going through this today, self build, low power, versus old servers versus what the F**k do I really need.

[@RaidOwl]

You don’t need a justifiable reason. If you like something and it’s cool and in budget then go for it.

[@nvbnvb2240]

Lithography E5-2699 v3 22 nm ...... Folks often think lets go for V3 we overclock it.....however v4 is a lot more power effecient 14 nm and pays itself off....even performance wise a v3 overclocked vs a v4 is not a big diffrence cuz the cache of the v4 is more efficient....and if you go dual socket ...power diffrence is multiplied bye 2

[@thewizardsofthezoo5376]

For taking advantage of those guys you need their very own motherboard, not something generic, there are too many extra functions compared to simple consumer grade chips. That mobo was indeed for gaming not for doing passthroughs with PCI.

[@nevernicemeadow]

i was on the brink of building a similar system. but as i live in germany with huge power costs, i yet dont need the number of cores, and pcie lanes, i therefore went for a 5700G, 128GB, b550 steel legegend combo, partly used parts, for around 600€ at that time. the benefit is, it only draws 30Watts (no discs) idle. if i ever neew more cores i ll swap the CPU for a 5950X or 5900X and then have like 50Watts idle

[@stephanejean2554]

im happy i watch your video before buy :)

[@dexsters5643]

Would of went for a single socket board and maybe V4 xeon or the 2699v3 xeon. With turbo unlock active to help with single core work

[@philmarsh7723]

Better to stay with the V4 xeons. More power-efficient.

[@Teksers]

PCI passthrough, ok but are you using it at it's full potential? Those CPU's have quad channel memory support but you only use it in dual channel. Also have you modded the BIOS for the CPU to always be at full turbo boost all the time?

[@rdsii64]

I wonder if this board will fit in that new silverstone 5u rack chassis you reviewed?

[@handymandan8494]

I just spent the last 18 hours trying to figure out why I can't get iommu to work. Vt-d is enabled, I have all of the settings in proxmox correct (I checked 153749 times) the HP Z440 specs say it should have it, the E5-1650 v3 says it should work, I don't get it. I feel your frustration.

[@IamTehGuyz]

This might be included in the "check settings in proxmox" you noted, but you added the right kernel parameters to grub for intel_iommu correct? I've a Z440 as well but with a 1650v4 and iommu is working fine and dandy. I also have the latest BIOS for the thing, in case that matters. Only BIOS setting you really need is VT-d, iirc

[@handymandan8494]

@@IamTehGuyz I appreciate the reply. I actually figured it out the next day. I had a different NIC than what I thought I had. New one arrived today. Going to give it another go tomorrow. I realized something was up when I removed the nic and the iommu number all came up. I felt silly, but its a learning process for me.

[@ashtoncable2480]

I recently tried to create a macOS VM under Linux using kvm and tried to passthrough my GPU. Only to find that the iommu groups weren't configured in the way I needed on my b450 board to be able to pass through the GPU. I was pretty bummed about that.

[@thudtheace]

Impulse buying is bad....I have 2 working Dell R715s', a Dell R815, and 2 hyve zeus servers... Bad impulse buy, bad! But the always on server is a i9-9900T with 128GB ram on an ASROCK industrial MB (It only uses about 15 watts, and up to 50 when abusing it), because the Dell's and hyve servers use a lot of power doing nothing. Cheers!

[@colinwilson210]

What was the bios missing, IOMMU? Not a clue what that does :-}

[@deyeatdapoopoo7582]

I think the main problem here is that you massively overspent compared to what would've been a much better deal which would have been to pickup a used workstation like an HP Z840 for $300 and spent less on the CPUs by going with something like 2x Xeon E5-2680v4 for $20 each instead of paying $125 each for the ones you got. That stil would've given you 28 cores/56 threads which is plenty, and you would've spent less than half what you did on your system while having it be easier and more straight forward since you'd have been upgrading a system instead of putting one together from scratch.

[@deyeatdapoopoo7582]

One other thing: the reason why your idle power use was so high is because that motherboard doesn't support max idle power states/savings. That's pretty common for Chinese motherboards, unfortunately. If you had a board that did, dual Xeons with 8 DIMMs and an SSD would pull around 80-100W, not 150W.

[@Fartowski]

Hello. I was thinking of building a dual xeon system to run unreal engine 5. How do you think the performance would be? Money is an issue, but I need something to use UE5. Thoughts?

[@calicomorgan2408]

I just found these processors for 25 Canadian dollars each and I'm specifically trying to find a motherboard that will NOT have this problem. I'm Supermicro boards seem to be the best option, but I've just started looking today.

[@mswork1764]

Just buy a used dual core z640 or z840 and you will get everything you need, including IOMMU, for well under $500 .

[@ericmartens8151]

I have a 2670v3 on a machinist chinese x99 board. Have had 0 issues in the year its been running so far

[@MWcrazyhorse]

I heard someone else building such a system as a workhorse power station. He ran into problems with his Nvidia cards not running. Further these systems seem to have issues with many drivers as they are designed for mainstream desktop configurations. Does it make sense to risk this or just stick with a powerful Am5 Ryzen setup? I just want the CPU computer power and some more lanes for multiple GPUs.

[@jarek0737]

I am having the same problem with a Dell system not supporting IOMMU.. I am trying to use a Dell precision T5600

[@msofronidis]

Xeons are not enough. You need the backplane busses and ECC compatible mofos to support it. Otherwise you end up up with an 8 litre turbo v8 engine retrofitted in a Camry. It will run but will eat the wallet you carry and a second mortgage and then ask for more and only give you a burned up pipes and an oil change with every supermarket trip.

[@msofronidis]

Xeons are not enough. You need the backplane busses and ECC compatible mofos to support it. Otherwise you end up up with an 8 litre turbo v8 engine retrofitted in a Camry. It will run but will eat the wallet you carry and a second mortgage and then ask for more and only give you a burned up pipes and an oil change with every supermarket trip.

[@ufukpolat3480]

I don't see how it's that power hungry. Haswell CPUs, as a feature of their FIVR (fully integrated voltage controller) are famously good with idle power. Even more modern CPUs aren't that good with idle power because they relay on traditional VRMs.

[@ezwider4652]

What operating system would you use on it? is there a reason I'd need microsoft server?

[@Freshiz325]

Well I guess it's time to get an Atomos recorder and capture to ProRes Raw from your FX3 ;-)

[@RaidOwl]

Lol I have one but it sits on my a7siii which is my main cam

[@YT_Creeper]

intel xeon for budget gaming: ✅ intel xeon for serveer machine: ✅

[@mikeccuk2006]

Old xeons are best test bench for containerized apps.

[@HardWhereHero]

I have about 10 to 12 ord 3rd gen pcs laying around. I already have the gaming PC's taken care of, as well as a HTPC but what else could i do with these systems. 3770 is not exactly chopped liver. I really need some ideas on what to do with them. I received 31 or so initially and removed the old OS's from around 20 of them and installed windows 10 and chrome on them. My dad took them down to the VA with him and gave most of them away. I think he still has a couple. So we even tried that route. I know I would like to make a server out of a few of them but not sure what to make.

[@sokmunky4104]

The problem is that you have to be very careful about buying Chinese motherboards without doing a good bit of research. Many of the no name brands are exactly what you said "Walmart like Motherboards". They work, but that is about it. Huananzhi are my go to boards when it comes to repurposed Xeon builds. IOMMU works great and I have never had issues with them. If you are scared of that then you can still buy very good used Supermicro E-ATX Dual CPU X99 boards off ebay for around $180-250 that will have all the features you will ever need.

[@josephololade]

There's a tool you can use to edit the bios file and add the iommu feature.. I can't remember

[@blackIce504]

youtube did that thing and went to next video despite i had it turned off, programmers suck now days. but anyway flash the bios with another bios of similar board or mod the bios as its clearly some code especially if they are using a proper X79 chipset from intel, as HW it supports it. to be safe with this make sure you have a hardware programmer just in case you brick your board.

[@Verzox]

thats the reason I went with a supermicro eatx board as my main test rig, 2011-3 so same cpu compatibility, metric fuck ton of ram, all the pcie you want, and ofc iommu and intel vt-d. all for 450$ which was mainly the price of 128GBs DDR4 2133 ECC RAM.

[@TruthDoesNotExist]

damn I just built the same system but only paid 16$ for the same cpu on ebay, the 18 core version is actually 30$ or you can get the 22 core for 170 and have 44 cores 88 threads, I also threw in 256gb ddr4 ecc, I built it as my main system and its so powerful so far theirs nothing it can't do. I bought the same trash motherboard an returned it and bought a different one that fit perfectly

[@tonynoel286]

Can you tell me what kind of mother board it is? Are you able to pass through a gpu to a vm?

[@TruthDoesNotExist]

@@tonynoel286 machinist motherboard, and I have no idea if I have iommu enabled, i don't need it anyway

[@tonynoel286]

@@TruthDoesNotExist Ok, thank you for replying!

[@jakevanoostrum1796]

E-ATX isn't really a proper standard, Most people and manufacturer refer to anything slightly bigger then ATX as E-ATX. I normally think ssi eeb when I hear E-ATX but the standard is different for every different manufacturer.

[@mswork1764]

Dude you should have bought a used z640 or z840 and you would have gotten everything you wanted, including IOMMU, for well under $500

[@andrewmcewan9145]

I thought jeff modded the bios to fix the iommu issue 🤔. I'll see if i can find the info.

[@RaidOwl]

I’ve seen some people do some bios mods on here but couldn’t find any that actually fixed the pass through issue.

[@RupertoCamarena]

im interesting upgrade my home server, i put all that hardware in my shooping list. question will it work with unraid?? thanks in advance

[@gastonhitw720]

instead of a chinese board i'm thinking about a dual supermicro board for 2 xeon 2699v3 hopefully

[@MacoCompany]

Why do you look up at the roof to the right so often? (your left)

[@RaidOwl]

Spiders

[@RealKeytones]

Just get an older rack server. For like $150 and throw the cpus in. Test pass thru.

[@BoraHorzaGobuchul]

Maybe I'll dust off my old e5-1230v-something supermicro-based home server and pay with it :)

[@NerdonWheels]

Yes. But what about the hicky on the left bicep though?

[@RaidOwl]

It’s a bruise…I swear…😳

[@thorium9190]

Would the Intel Xeon E5-2699 v3 also work? Giving a total of 36cores and 72threads

[@tom_w67]

with the Chinese dual socket motherboard i think the x79 version is better because it has a 602 chipset that support pci pass-thru...i think???? the x99 version doesn't like you found out.

[@RaidOwl]

Yeah I think you’re right

[@joeyjojojr.shabadoo915]

While synthetic benchmarks are not exactly representative of the overall system's capabilities, your GeekBench5 score did pain me a bit to see (758 Single/16064 Multi over 32c/64t), as my Ryzen 3950x 16c/32t CPU with minus 50% of the cores, but with a bit of 'tuning' (Under-volting & P-State manipulation using Ryzen Clock Tuner, or CTR) my 24/7 tune sees a GeekBench5 score of 1393 Single/15950 Multi. It also (as you know) supports ECC Memory (not cheaper Registered, but still) on x570 Chipsets and the CTR app is the reason I decided against a 5950x upgrade, as my CPU seems to still be a 'contender' for what I use it for. The above combined with the fact that it supports all passthrough functionality is the reason it will end up in my Rack eventually. All it needs is an IPMI solution (PiKVM for example). I think the days of old repurposed Xeons vs Ryzen desktops setups is a conversation worth having, with the only 2 downsides obviously being IMPI and Registered DDR4 lacking....but 1 of those can be solved. I have deployed my LAST Xeon Home Lab box (for compute anyway, as Disk shelves will remain the exception). I think your video is great information, if only for the power requirements and physical oddities. browser.geekbench.com/v5/cpu/12804832

[@trumpsextratesticle8590]

"Im disappointed that your stock CPU from 10 years ago didnt beat my Modern over clocked , manipulated CPU." Dude, shut the fuck up.

[@michaelcollins4458]

could have gone with Fractal Design Define XL R2 it's slightly cheaper