What the NUC Kind of Cluster is This???

[@jcugnoni]

Reminds my several 'Beowulf' clusters for scientific computation that I built 10+ years ago by stacking miniITX small for factor I5-3450s (4core 16Gb ram each) and hooked them to a gigabit switch. Node0 was running Ubuntu with storage pools, management tools and the other nodes were booting by tftp with all the files hosted node0. DRBL linux package was handling the whole boot, os config of the nodes, network mounts magically. This cluster has served us well for dynamic fluid/structure simulation using MPI for more than 7 years and was totally unbeatable in perf/dollar. Maybe I could do that with those mini pcs. With the current CPU perf / watt and core count this ciuld be great..

[@ktfjulien]

@Wendell TRY THIS ^^ !!

[@DigitalJedi]

I used to have a single big box with a 6800K and dual Titan Xps for simulations, but I remember the computer science lab in my university having a stack of 1L office PCs with i5 3450Ts. IIRC there were 3 stacks of 6 each for 18 nodes or 72 CPU cores and a combined 144GB of RAM. It was really funny hearing them all spool up their fans. You could tell how hard somebody was hitting them by how many of them spun up.

[@autohmae]

At some point HP released Open Single System Image (openSSI) Linux Cluster Project, which was also kind of interesting, but sadly didn't seem to go anywhere.

[@bosstowndynamics5488]

I'd love to see some more stuff specifically on hyperconvergence itself, what software supports it (XCP, Proxmox, eventually TrueNAS etc), how to use it, when to use it and maybe even using it with mixed hardware setups for the homelabber with a need for more PCIe lanes than these mini PCs can provide.

[@Foiliagegaming]

I’m in the process of building a new lab and I have been struggling on what to get. This has helped me a bit. Thank Level1techs!

[@echtogammut]

I bought my mother one of the MinisForum "game" PC's to replace the MicroITX computer I built her a couple years back. For the money, I couldn't build something better and certainly not as small. I was really impressed with the little thing.

[@prashanthb6521]

When it breaks you cant even run around to get replacement parts. While a desktop computer can be repaired easily and cheaply.

[@bcredeur97]

I had no idea thunderbolt could be used for networking like this. This is genius!

[@autohmae]

USB also supports networking as does Bluetooth.

[@physx_yt1062]

@@autohmae How would you use USB?

[@autohmae]

@@physx_yt1062 for this application ? I wouldn't maybe USB4 is good enough. Just saying that all of them have IP support.

[@lunalucadou877]

It feels like the logical endpoint of Thunderbolt. Thunderbolt is PCIe, which is basically a packet-switched network, so why not use it as the carrier for another packet-switched network (IP)?

[@TechnoTim]

Cattle not pets! Build availability into the solution, not into each device! Love it!

[@AlexSchendel]

I personally would love to see more Thunderbolt (or USB4 for that matter) adoption in desktop motherboards.

[@tames307]

Thunderbolt adoption was slowed by Intel's exclusive-use deal with Apple when it was first released. I'm surprised that it hasn't taken off since then.

[@StaySic4Ever]

Yeah lack of USB4 even on expensive boards is simply horrible, to add just regular type-C adoption is also horrible, everyone slacking.

[@Decenium]

ehhh idk, we are using docks these days at work for employees with their usb C connection points and honestly... I thought usb C was suppose to last long but it kinda sucks, they become so weak, never had that with usb A or B.

[@potatoes5829]

@@Decenium agree, had a couple usbc ports bend in ways usba never would

[@LampJustin]

Yeah, the video I've been waiting for ever since NUCs popped with two TB3 interfaces 😊🎉 Real sad, that the latency is that high though. That makes it totally unusable for replicated storage...

[@upgrayedd251]

So happy to see another happy outcome! Your hard work is making a difference!

[@lostphotographs3936]

Wendel looking good Sir! Great content from the team. 2.5 G in a NUC is a win.

[@physx_yt1062]

That's pretty cool, no added power consumption and heat from extra networking cards and for a cluster like this, the limit on the cable length is not an issue. I would be interested in the possibility of using a thunderbolt 4 hub to take all of this to the next level, but that would add cost and complexity and I suspect that a hub wouldn't work with multiple hosts in the first place, as that's not what it was designed for.

[@Warrigt]

Useful info amongst the rambling

[@gmanyyavailable]

If you do eGPU you'd know that Thunderbolt's 40 Gbit number is a marketing term. It can only do 40 Gbit in DisplayPort mode. In PCIe mode it goes down to 32 Gbit (four lanes of PCIe 3.0), but somehow this is before protocol overhead. After protocol overhead, you get maximum 2.2 GiB/s transfer speeds very best case scenario

[@mikegrok]

I have a Genoa 9124 with an air cooler. After boot is is actually very quiet. The cpu has a max power of 200 watts, but the chip area is enormous, so it has no trouble getting rid of the heat.

[@alexlovett1991]

Love this stuff, I looked into it a while ago but with IP over USB3 which is a very accessible connector (a lot of homelabbers don’t have modern thunderbolt systems but old laptops / mini pcs). Would love to see if that’s possible (haven’t got round to trying it myself).

[@benjamintrathen6119]

I have often wondered why there is no daisy chain cable? For nucs with only 1 thunderbolt port this would be nice.

[@MarkRose1337]

In the days of old, I ran a NUC Now I'll need at least 3 those when I muck Cause I do a ping-NUC 1, NUC 2 Then I save a NUC for when there's nothin' else to do

[@GabrielFoote]

That in inconsistent ping is um... concerning

[@benjamintrathen6119]

Yay finally the NUC cluster, I have been looking forward to this since you mentioned in another video

[@marcin_karwinski]

Great thing for a K3OS or similar Kubernetes cluster on bare metal OS...

[@Toddimus831]

Would this work with USB3.2 (gen2x2,or whatever the 20Gbit version is called)? Even if it’s not explicitly called Thunderbolt? I’m assuming it would definitely work with USB4.

[@rpm10k.]

Thunderbolt is not USB.

[@jonny393]

not been here for...a year, maybe two... What happened you look good man!

[@DarkShadowReign]

He got bit by a tick and got sick and had to change his diet.

[@MyAJVideo]

FYI old lenovo micro pcs have PCI so you could add a network card :D

[@WilReid]

I wonder, and I know it sounds crazy, if introducing a TB to PCIe chassis with a proper NIC would work better or just increase the latency and overhead. It could show if it's the interface or entirely the driver.

[@MHTHINK]

Would thunderbolt provide the lowest power 40Gbps connection? Certainly less than adding an older sfp+ card..

[@Level1Techs]

mostly no, closer to 20gbpa

[@MHTHINK]

@@Level1Techs Thanks for the reply. I'm just very interested in power consumption lately, now that I'm living off grid. My old-ish enterprise strategy is no longer cutting the mustard.

[@BryceDearden]

If you only end up getting about 10Gb anyway you could use that GoWin R86 that has two 10 gig SFP cages on it already.

[@Gerben_Meijer]

How about bridging the thunderbolt interfaces on each nuc, and running spanning tree. Now you have a flat br-thunderbolt network with just one ip per nuc.

[@0LoneTech]

Spanning tree protocol essentially works by logically cutting links until the network has no loops. You'd end up with one node handling all the traffic between the others, even though they have a direct link going unused. That also adds the full latency to all the forwarded traffic. Running one IP per NUC is fine, as long as the routing is correct (multipath for redundancy).

[@LeminskiTankscor]

Oh I'm not sure. It sounds like a setup for a complete Clusternuc to me.

[@djstraussp]

Nice idea for 5 node lenovo m90q mini cluste, but using 10gb and no switch. This could save a lot of money. I think this could be done on plain ethernet NICs.

[@chrisg9215]

Also is the ping introducing significantl latency that manifests when using the cluster? I need to do more home labbing....

[@gustersongusterson4120]

In my opinion/experience, thunderbolt was a promise never even close to being realised. I'd avoid the headache personally. Maybe more mini pcs will start being designed with SFP ports which are slower but without the bs.

[@RyTrapp0]

Like so many Intel developments over the last decade or so

[@feralshad0w]

How does the ngenius system compare to the Linksys omada?

[@mikedeath7333]

ask eta about the gamecube and think makita toolbox plz and Thank you

[@OCONTECH]

400,000K SUBSCRIBERS WELL DONE TEAM @ LEVEL 1

[@ewenchan1239]

Silly question - how do you install a PCIe 3.0 x16 card into any one of these mini PCs???

[@buk0wski]

I think I may have missed it, but what's with those ridiculous pings (on a local network??). Is this just a poorly written driver?

[@AzureFlash]

15:57

[@microcolonel]

Now that intel is discontinuing NUC... do we now call them PUC?

[@MarkRose1337]

I''m surprised this wasn't on the Linux channel

[@CataclysmZA]

Level1Linux probably does not have the algorithmic reach that the main channel does, so this will do better here.

[@LA-MJ]

What I am hearing instead is don't bother with TB networking unless you have nothing above 2.5Gbps

[@FrenziedManbeast]

What's my lowest-wattage and cheapest cost to having a home machine with: ECC RAM, 10GbE, Intel Quick Sync, and at least six cores/12 threads?

[@marcogenovesi8570]

still pretty high, you are looking at Xeon D boards

[@FrenziedManbeast]

The Xeon chips aren't APUs too, right? So I'd need an Intel GPU w/ Quick Sync support as well? Seems like a lot of PCIe connectivity to give up just to gain Quick Sync, that's a shame.

[@bosstowndynamics5488]

​@@FrenziedManbeastIntel does make some consumer chips with ECC but you'll probably not get 6 hyperthreaded cores (they only do this on low end chips where there's no competing Xeon), so you'll need to decide if 4 high performance cores will do compared to say 6 older generation, more power hungry cores. Do you need QuickSync specifically though, or just hardware accelerated encoding? Because AMD's 7000 series will run really efficiently when underclocked slightly, is readily available in 6 core configurations for *relatively* cheap, they all support ECC and have onboard GPUs so they *should* be able to do hardware encode (do double check this first but as long as the encoders are there AMD GPUs are generally well supported on Linux).

[@FrenziedManbeast]

@@bosstowndynamics5488 I should have specified that I'm looking for fully Registered ECC. I'm still curious as to what the lowest Watt + lowest cost solution would be. I understand that many consumer CPUs support ECC on the data sheet, but finding motherboards with support for Registered ECC for those chips seems...difficult? Impossible?

[@MarkRose1337]

Many i3s support ECC RAM.

[@mikedeath7333]

1tb of cache? would that help an os #3d cache clockfriquency issues(handler prioraty)?

[@ahyi9350]

While running this in cluster with 2.5gbps, doesnt it get slow?

[@ChrisOfSDUB]

Now do it with firewire

[@WarkWarbly]

Is there an active occulink m.2 card that would work with this kind of stuff?

[@Jango1989]

Love this!

[@VideogamesAsArt]

This is incredibly cool

[@DavidAshwell]

"Soon" showed up!

[@jyvben1520]

or go retro, use firewire (400/800) was faster than (in theory) than 100Mb, but also only point to point.

[@kevinclark1466]

Can this work with USB4?

[@noname-vl6vy]

can you do 5 node on that setup? please?

[@CataclysmZA]

Three node is a practical limitation because there aren't enough Thunderbolt ports available.

[@0LoneTech]

​@@CataclysmZA Only for full interconnection. You can still run a ring topology with two paths between any nodes, though forwarding through networks with this poor performance isn't going to be fun.

[@TheExard3k]

Those latencies (ping) is unacceptable for using Ceph. That's worse than a WAN connection. Sequential transfers should be fine, but for clustering, that's a no go :(

[@haakoflo]

So out of the box we get 1Gb and 100ms ping? Or we could use the 2 2.5Gb interfaces for 5Gb aggregate, and 0.1ms ping?

[@snowwsquire]

2x10gigabit for a little setup when you first get the nucs vs 2x2.5gigabit oob is an easy choice for me, that 10gigabit is gonna be much better for me

[@marcogenovesi8570]

If you are not using the thunderbolt it's not worth the 100-150$ markup over similar devices without thundetbolt. Also if you use both ethernet ports to do this then what network interface is used to talk to the outside world?

[@tcnymex]

​@@marcogenovesi8570 perhaps talk to the outside world vua a wifi card in a slot where a 2nd ssd would normally go

[@ShroomheadOne]

This video title is far cleverer than it has any right to be.

[@SaltCollecta]

The Minis Forum Quorum 🥳

[@boydfields]

You're just great!

[@zushiba]

First, I finally got some internet points!

[@cairath]

Cheers ;)

[@marcogenovesi8570]

all the points, give them to him/her/they/it

[@V1N_574]

I just imagine youtube sending firsts certificate

[@tommybronze3451]

We still have a situation where a low cost nvme will saturate ethernet socket becuase those mini pc's insist on not using 10gbe :/

[@greygag36]

Couldn't you bridge the interfaces and assign an IP so it's the same IP space. Simpler config with the same design.

[@iodehk5205]

P2P is better. STP bad

[@chrisg9215]

I'd leave a positive comment but I'm all outta sorts without my morning new... Links with friend sharing...... I agree with guy who said we need Krista to fill in with Wandall and Ryu puppets....

[@shambleslongplay3566]

Why no level 1 news.. I mean level 1 sharing links with friends??

[@OGSkumse]

So basically iSCSI, but with extra caveats? :D

[@awetisimgaming7473]

I would actually like to put 8 orange pi 5s in my system that has a 7950x, and I'm wondering how easy this setup is gonna be with the way people talk about it. This seems somewhat daunting, and I may also opt to use the m.2 slots for interfacing instead of type c or on board ethernet

[@stuffinfinland]

Unable to compute.

[@mikedeath7333]

Bisides solar bitflips cancer is the main catalyst

[@grantwiersum7394]

The Jetson nanos can be clustered together to build a mini gpu cluster. Beastly for inference and more than capable for training small NNs. I've toyed with the idea of building a luggable/cyberdeck with 5 nodes and an ebike battery in a briefcase.

[@gillianseed4419]

software networking, I mean it can work but its gonna suck down half your cpu cycles and run half the speed

[@mikedeath7333]

what is death? a blesing

[@sregister]

Anyone actually run iperf or netperf over a thunderbolt-net interface like this? In order to do highspeed ethernet a lot of driver/sw tricks and hw offloads are needed to run ethernet efficiently and im not sure if just raw thunderbolt is enough but maybe it is

[@mikedeath7333]

3d vcache vgpio? pathrought? think capturecard but quantom arm meets x86 #northbrydge (threadripper ipad pro brydge keys?

[@Iatimere]

what the NUC?!?

[@samiraperi467]

ClusterNUC.

[@erk5243]

DeeeZ NUCz

[@mikedeath7333]

Pcie turning pi 2

[@mikedeath7333]

risc-v cores can run an os at 1mhz zo can i volt it up to like 120dc and have the chassis run on 24v #capasotor pack+ discharge tool arcing prev.

[@midnightlaundry2800]

Any dude in I.T. named Wendell has to be the man! LOL

[@mikedeath7333]

1tb virus or "Ai" xD #uranium forge eeprom updater

[@crlshn]

Jajajaja thunderbolt en linux y con AMD or nvidia hahahahade, please don't. Yet. Early adopted back & no Bueno

[@mikedeath7333]

Escape from tarkov cheatercard fixes vigpo?

[@brianhansen2202]

I wish this guy would learn, that the background music is very annoing... Otherwise, I love his videos. I'm not interested in music, when watching computer-videos, and when I want music, I prefer to choose it myself