Hardware RAID for the fastest Raspberry Pi CM4 NAS!

[@JeffGeerling]

Like if you want to see Jeff get shocked more (- RSJ).

[@badrulhasan9935]

ok

[@Dygear]

ElectroBOOM liked this message, lol.

[@the_beefy1986]

Maybe just red-shirt jeff

[@juri14111996]

Hardware Raid is great, exept: - good lock if the controller fails and you cand find the exact same model - no easy way of getting Drive fail allerts to the os. - it expensive

[@jayrowe6473]

Sure, why not?

[@Centorios]

props to the broadcom guys that were just as hyped as you to make it work

[@vincei4252]

Props to Broadcom. No, seriously, big props. Thanks, Guys!

[@drdiesel1]

Agreed, hope the necessary changes get mainlined as I'm slowly moving more and more systems to ARM.

[@JeffGeerling]

The engineers were just excited to see if they could get the thing up and running, and they did! I'm so grateful they sent the card my way, I had all but given up on trying to get hardware RAID working. Now, if only someone at AMD or Nvidia would lend me a driver engineer for a few minutes...

[@fahrimertdincer8421]

@@JeffGeerling XD if this happens can you sent nvida or amds driver to us ı have same nvida card on my pc and ı want to try some thing on it at least windows to run proprely

[@richards7909]

I was about to say the same thing as Vince. It’s a good thing to see :)

[@vincei4252]

@@richards7909 Totally. I was in a place where my goto is a corporation will bend over backwards not to help, ignore or dump lawyers on you.

[@mme725]

Give a Geerling a PCIe lane, and he'll run a mile! Once again fascinated by your process and thankful for you sharing your efforts! It's crazy seeing how far you can go with your Pi!

[@AndrewDanne]

I just loved this session. I use to run a Lab for XFS & CXFS at SGI in Melbourne Australia with petabytes of storage and more than 2000 CPUs. The Broadcom guys are the best as we were forever rebuilding kernel drivers on beta cards from Broadcom (also with networks). Broadcom was always the most helpful and knowledgeable of all the Interface people we worked with. You brought back some great memories of working at this level of development. Thanks for the work you do, it's hugely interesting.

[@DanCave]

G'day Andrew.. not spoken for ages.. :) good vid... #SgiAlumini

[@shaigluskin1225]

Let me be clear, I have no intention or need to try this at home. What I love about this and Jeff's other projects is witnessing his incredible patience and tenacity. Recompiling the Linux kernel 50 times! Go Jeff. I smashed the applause button and chipped in.

[@JonathanZigler]

Sir, as a firmware engineer I had to order your merchandise.

[@JeffGeerling]

Release the magic smoke!

[@JarrodCoombes]

@@JeffGeerling You know, I'd probably buy a shirt that said "Until next time, I'm Jeff" :D

[@bravosixactual3000]

The Virgin Linus: 24SSD tHreadRiPEr server The chad Jeff: haha raspberry pi go brrrrrrrrrrrrr

[@zambonidriver42]

Group buy 200?

[@CyberBlaed]

And the chad got a working/better raid driver, legend!

[@SpencerYork1534]

Dude I'm jealous this sounds like a great time, getting Broadcom engineers involved in the project.... greatness :)

[@playeronthebeat]

I'm so stoked about that since I want to build a NAS out of the CM4 as well! I desperately need it for my photos as they're taking up like all the space on my hard drive (and I've got no copy... So, if it fails, it's all lost...). You're doing such a great job of explaining and showing off all these cool things! Thanks, man!

[@Miland3r]

"Required about 50 kernel recompiles" ~ You should join us in Gentoo land, you'll feel right at home.

[@N3tech]

I’m finishing up my last semester of University and I’m building a autonomous robot system with a Pi for my senior design. Everything you say is so many levels beyond my understanding / capabilities but you do such a great job and explaining and keeping it interesting. Keep up the great work man!

[@JeffGeerling]

Every day the things I do are beyond what I did yesterday, just keep learning and growing! Last year did I think I'd be debugging storage cards over a serial UART connection? Nope! But now I can say I have, and almost every time I do something new (which to some may seem basic, to others voodoo magic) I realize it was only intimidating before I tried doing it. Most of the time it's a lot simpler (conceptually at least) than I feared it would be!

[@joshsinykin5230]

@@JeffGeerling Debug is art form learned over years of tinkering. Its not something you get immediately out of college. If you continue to ask the question, form a hypothesis, find a test to prove or disprove you can debug anything.

[@antoniovargas3969]

This is an absolutely ridiculous setup and use of hardware. Yet, it is exactly the kind of thing I would do and loved nerding out watching it. Thanks for putting it together.

[@tdtrecordsmusic]

wow , there are sooo many accolades here !! Going forward with opensource !! Working with Broadcom shows SOO much 4 both parties !!! Keep going Jeff. This is all gr8 !!

[@ItzNickyJ]

I legit am not even interested in anything ras-pi related but this man... This man has my interest. Good work sir.

[@paulmaydaynight9925]

that was fun :D top marks for the Broadcom engineers

[@syntheticperson]

I love the quote “released the magic smoke”! 😁

[@rishabhtomarA7da6]

"i hate to say mibibyte" This line hit me hard.

[@NeemiasJunior1]

I have very little experience with hardware but watched til the end, amazing video man! I've got 10% of what you said but motivated me to dust my pi3 and study more. Keep it up! Cheers

[@AshutoshJatkar]

Thanks

[@DudeSkinnyTall]

Ok guys, now we've got to hit that 1M subscribers mark...

[@PRAGEETHKARUNADHEERA]

Thanks Jeff. Love this sort of work. You inspire me to move my CM4 SATA nas to SAS.

[@brycejeannotte7699]

Wow, that was a lot of work to get that going. Great job. I am looking forward to the availability of the multi SATA cards, the RAID configuration you put together is over-kill for me.

[@marcg1043]

Awesome! Sas is the way for pi NAS. The used enterprise equipment is so cheap and rock solid. Jeff thanks for linking the two worlds.

[@joshsinykin5230]

This is the way :)

[@christopherrasmussen8718]

I love this stuff. I do it, but I have a Buffalo NAS in the rack. I can't believe Broadcom sat on the phone with you like that. I worked with enterprise SASs in the DOD. Hugh petabyte things. Just the idea of a small little box on the desk with superfast multi terabyte storage RAID. Crazy. I do have a few RAID cards from dell and one from Promise (fastTrack) I going to play with.

[@TwistedMe13]

4:05 There is also a third reason you didn't mention but probably fits given your passion for experimenting with the Pi boards and other stuff You're setting the stage for future Pi/CM boards by developing the groundwork now. Speaking of Broadcom's interest in this beyond doing a solid to major developer for the Pi platform they've established is that currently the CM board's PCIe capability is too anemic (the card they sent you is capped to 1/16th of its nominal bandwidth right from the start) for serious commercial interest but for the price of that raid card and the man hours working with you to troubleshoot it-- it lays the groundwork for when the Pi CM devices they create are capable enough to truly power a fully loaded SAN they already have most of the kinks worked out.

[@JeffGeerling]

You had me at the kinky fully loaded SAN!

[@austinfarley4971]

Hey Jeff, IT guy who is a network and systems administrator.I feel like you need to add a disclaimer about RAID in your videos. RAID is not a replacement for a backup unless you don't care about your storage. Outside of that, I love your videos! Trying to get my hands on a bunch of PoE HATs and a rack tray to put them all in.

[@alliejr]

Synology and QNAP have been selling ARM-based NAS RAID forever. Except for sotware RAID, such storage boxes have never been CPU-bound. But, yes, a fun project and great video.

[@SorenEragon]

@Jeff_Geerling I love your content, but today I want to thank you for posting your bloopers. It gives me a good chuckle and is slowly getting me over my trepidation about posting and streaming content myself ^.^

[@DrorF]

I always wonder who's he gonna be next time, but it's always Jeff Geerling... Which is great.

[@DevOpsDirective]

Great as always! Looking forward to the more affordable version!

[@HaraldurBaldursson]

This is a potentionally interesting bundle to buy ready and package hw + sw fully ready

[@hrnekbezucha]

That's quite a hobby you got there, mate :D

[@o_o6869]

broadcom and jeff need some love.

[@PeterMountUK]

I really need to get back into trying to get Fibre Channel working with my CM4 to see if I can connect it directly to my SAN. I've got the card detected, that was the easy part, but it's now stuck with kernel modules. They compile but the HBA doesn't initialise on boot so something is missing, just need to figure out what

[@Chris-rm1pn]

The raid card that has same amount of ram that pc has 😂

[@JeffGeerling]

RAID card is flexing on the Pi.

[@haydenc2742]

@@JeffGeerling LOL...Raid Card (in German body builder voice) "Your a girly man computer...I'm here to ( *clap and point* ) PUMP YOU UP!!!!"

[@martyburgess341]

The outtakes are a nice touch and funny

[@BeefIngot]

I love how specifically he mentioned the potential benefits of raid. "And not lose access to your data immediately" This guy "raid is not backup"s

[@wikingagresor]

If someone at Broadcom is watching this, then let them know that this is a golden business opportunity for them: pairing some card with a board like raspberry into a thin and silent case with 8 slots for disks and you will have a 'mini nas' for home use or some small firms, where power consumption and silent mode of working is more of a priority than computing power.

[@joshsinykin5230]

Message received.

[@hexd0t]

16:03 please don't use anti-ESD-bags to run electronics on them - they are designed to not let static build up, but are NOT good insulators, so you can short pcbs laying on them.

[@JeffGeerling]

This is true. I really, *really* need to 3D print an enclosure for the CM4 IO Board that allows me to physically mount a PCIe card above using a riser.

[@notsonominal]

Yes, they need to be conductive to be anti static, but the resistance tends to be relatively high so 5v is unlikely to be an huge issue - still its bad form.

[@CrisanBogdan]

@@JeffGeerling this would be a good video topic :)

[@eeandersen]

Ha! I saw that too! Reminded me of my own bad habits - hot on something and needed an insulator quick...

[@AnotherNerdHere]

Jeff, I think you are Sir Edmund Hillary of the RaspberryPi world. When asked why did you do it? "Because it was there". I love watching you do all kinds of interesting stuff. Thank you. I had to laugh at how do you tell its Enterprise grade comments.

[@shizzyrizzy]

Good job! Enjoy the BAT! Keep up the good work!

[@JeffGeerling]

Thanks! That was you then :) Always a welcome surprise!

[@jlleung999]

Thanks Jeff, this is exactly what I am looking for. (just found it's a 2 years old video)

[@huguia]

You can start a PSU without a PC if you short the green wire on the 24 PIN connector with any of the black ground wires. Great vid as always

[@JeffGeerling]

That's what I did :) The problem is, one of the other wires I had cut off on the connector shorted out. Should've removed the wires entirely instead of leaving them dangling.

[@neiljones2761]

@@JeffGeerling maybe you should invest in something like this www.amazon.com/CRJ-24-Pin-Switch-Jumper-Sleeved/dp/B01MSY4966, or make your own.

[@brianbutton6346]

Holy cow! What a story! I laughed, I cried. Better you than me.

[@cll1out]

Impressive getting such high end enterprise hardware to talk nicely to a Pi

[@hajajmaor]

Great channel, hope to see more rpi projects

[@Squinoogle]

What we've all learned here is that it's fun watching Jeff try to say Mibibytes 3 times fast :D

[@PartySlothy]

Hey Jeff, really interesting project. The first thing I thought is how you are gonna get the speed from the raid array over the network since you already used the pcie port for the controller and the lan port is the limiting factor. In case you want to try this with a cheaper setup maybe you can go for a cheap SAS2 expander like the BPN-SAS826A and plug in the drives standing directly - it's quite jank but might do it for a prototype. Regarding the atx PSU, I rigid one together by shorting the green startup (5v VSB+) with a paperclip or short wire. You can use a cheap used OEM PSU like the FSP350-60 that comein at maybe 20$ and of decent enough quality. Anyways, I'm looking forward to your future shenanigans on this project. Keep up the good work!

[@tenminutetokyo2643]

Back in the day we used to have customers RAID 5 Zip 100 disks together.

[@avejst]

Great job 👍 Thanks for sharing your experience with all of us 👍😊

[@alextirrellRI]

Wow -- this is actually something I was recently looking into but couldn't find much on. I have been using a Windows-based machine for HBA SAS RAID but I couldn't justify the power usage. I just bought a used ReadyNAS because I wanted something that used less power. Sadly it can't use my SAS disks.

[@hokuspokus8570]

Waiting to see progress and fitting everything in one case.

[@JasonLeaman]

Wicked video, thats the exact card i have been looking for on Ebay for my Lenovo St550 server :)

[@JeffGeerling]

Server pulls FTW!

[@JasonLeaman]

@@JeffGeerling I so need one !

[@nicolaslavinicki4029]

I'm excited for part 2!

[@StripeyType]

I'd love to know more about that backplane/drive cage

[@edwardallenthree]

I saw the same backplane on Linus' chanel. Broadcom must use them for internal testing.

[@homecontrol9034]

Hell yeahhh !! good work jeff ! go on id like to see more crazy stuff like that.. greetz from an ITGuy

[@LostInThe0zone]

Looks like a fun project.

[@mechbear]

That comment about r/homelab just made my day

[@berndeckenfels]

Besides cpu parit calc a hw raid also saves PCI bandwith as it transfers only one write replica

[@hectororestes2752]

Hi, would you mind expanding on the "transfers only one write replica" part? im trying to make a physically small server using a tiny (1 liter) form-factor pc that has a single NVME slot (pcie 3.0 x4 lanes) so im researching what is the most I can do with it.

[@berndeckenfels]

@@hectororestes2752 well if you have only one disk then this does not apply. But when you do software raid then each write will be written two or more times (to each data and parity disk individually). Those writes pass the PCI bus multiple times. If you on the other hand use a physical raid then the write is written only once to the HBA and that will write it (ideally over separate channels in parallel) to the attached disks without needing any additional bandwidth on the Cpu PCIe lanes.

[@coffeemaddan]

Amazing content. Well done!

[@ninline2000]

This video highlights the huge advantage the Pi has. A cheap SBC with support from people like Broadcom. So many cheap SBC systems get next to no support.

[@JeffGeerling]

I actually asked the engineers about it-the teams that work on storage (like the card and backplane I was testing) has very little relationship (AFAICT) with the wing that manages the ARM processors. It's like that in most large corporations. Broadcom Inc. has a very interesting history (the storage cards went LSI, then Avago, then Broadcom, through a series of buyouts/mergers... and I'm probably missing a lot of this history there!).

[@joshsinykin5230]

@@JeffGeerling you got the history right from 2000- now. The LSI adapter you originally used was from the same business division that is now Broadcom.

[@EduardoRubioLogan]

The closing section of the video answer ( kinda) my question, What is the lowest price for a complete setup ( minus the storage) that we can build? Nothing fancy, but that uses the pi and controller efficiently. Like this controller card can manage 3.2 Gbps speeds , but the pi can't handle that Awesome video by the way !

[@jensdroessler3575]

So, now YOU got that card and that prototyping drive case!

[@williamhart4896]

Now all we need is a inexpensive 64bit octacore arm that's got the guts for a decent desktop a m.2nvme socket and dual soddim ram sockets and decent ports and costs . The 4b is almost there otherwise excellent test Jeff .

[@coreforge]

I haven't had issues with my powered riser (PCE164-3N03 VER 006) so far. Since they typically use the shield of the USB cable for ground, that could be the issue if you're using seperate power supplies (I believe you said you had better luck with a thicker cable from a different riser that with the thinner one that came with it). I'm powering the pi and card with one power supply, so there is a good ground connection that way (an old 250W ATX power supply that still has floppy power connectors for the cm4. I also just put a bit of 1mm solder into the 24pin connector of the PSU bridging the green wire to ground. Older power supplies might also need a base load to give out a stable voltage, a power resistor of brakelight works pretty good for those. Newer ones shouldn't need it though.)

[@JeffGeerling]

Thanks for the info-the risers have been really hit or miss, but I do wonder if the ground and the separate power supplies could be the issue. I might try powering everything through one next. I just wish someone made a proper modular PC PSU that was intended for 'bench use', with a bunch of molex, PCI, etc. connectors and a switch, and didn't rely on PS_ON to turn on!

[@coreforge]

@@JeffGeerling You can get those ATX breakout boards on ebay that often have a switch for PS_ON for fairly cheap. They only need the 24pin connector, so the other ones are still free.

[@AJMansfield1]

In theory it's actually be possible to get _better_ than just PCIe 1x speed on a PI NAS, if you use a multi-lane switch and can set up the storage controller and NIC to talk to each other directly with PCIe DMA.

[@JeffGeerling]

True, and I was actually talking to the Broadcom folks about this... it might be possible to get a full 3-4 Gbps read/write if keeping it through the PCIe cards, bypassing the Pi's CPU.

[@AJMansfield1]

@@JeffGeerling also, here's a rather ridiculous application of this I just thought up: if you do get PCIe DMA working between the Raid and NIC, then you should be able to connect the pi's own NIC to the PCIe NIC and aggregate that 1Gbps with the 4 Gbps through the PCIe to enable the pi to access the storage raid at up to 5 Gbps rather than just 4. (Actually, if it's possible to configure the pi's integrated NIC to talk RDMA, this might actually be feasible to do, just barely.)

[@dtb9165]

I have been testing cards made with the old PCI tecnology, what really impresses me is that a PCIe card works on a non-x86 computer. Very interesting.

[@eden1925]

You can jump start an atx psu by shorting pin 13 and 14, there are diagrams online too.

[@jonshouse1]

Buy a cheap "PSU tester", they have a few LEDs for the power lines and a short to power up the ATX PSU. Great for quick testing PSUs or when you need get high current 5 or 12v without magic smoke.

[@danielhoffman3802]

Awesome work. Loving all the storage stuff on a rpi. Are broadcom selling that universal backplane as it would solve so many problems for my home storage needs.

[@JeffGeerling]

Unfortunately it's a 'reference'. Other vendors will likely start producing them, but it's still a new-ish standard and there are only a few servers on the market with it yet.

[@abx42]

Damn that's some load AF!!!! The drives sound like the caterpillar drive from Hunt for Red October and how many times have you compiled the colonel lol

[@deno1122]

Really cool Jeff

[@frauseo]

You cracked me with the red shirt jeff blowing the psu 🤣

[@JeffGeerling]

Sadly, that was just normal-shirt-Jeff :(

[@ryanwakebradtelle8682]

I forget what it's called but the reason your hard drives move the entire case, is the same reason that metronomes sync up. The the small forces average out over the entire device, this is why you need Enterprise grade drives if you're going to have more than eight drives next to each other.

[@first-thoughtgiver-of-will2456]

You know I don't know much about RAID in terms of hardware and software but if I was hired to build a distributed data storage solution I would strongly consider something like the raspberry pi (with redundant power supply like feature and maybe a little more RAM) due to the ability to apply rolling updates, manage caches such as metadata on NVME and highly requested transactions on NVME as well as compression and error correction encoding parity. I would probably use Rook on Kubernetes to implement Ceph (admittedly this is the only storage setup Ive procured myself but was and am thoroughly impressed). The Kubernetes cluster could even be scheduled for other jobs dynamically if storage isn't being accessed a lot-- try doing that with a hardware raid setup.

[@izzieb]

You've got a lot of SAS in this video. You've been hanging around with Red Shirt Jeff too much.

[@JeffGeerling]

r/dadjokes :D

[@izzieb]

@@JeffGeerling but I'm not a Dad... Unless... Ah crap

[@bennyfactor]

14:02 would you say that it is a ... fully ARMed and operational data station?

[@nbtmx1]

Or is it ARMed and SASsy

[@z185284]

13:57 My Proliant drives do not make that noise, might want to check into that.

[@bhume7535]

Yeah, mine came out of a dumpster and don't sound like that. definitely not good.

[@playdoh1975]

What hba enclosure were you using??

[@Airbag888]

This is exciting and eye opening.. all the assumptions are out of the window like, the rpi4 can't handle the full speed of the gbit in NAS use cases even if the IO is doing over 300MB/s Very cool that you reported the power usage at the wall as well. It's obvious the i3 with their ATX psu is going to be extremely inefficient at these low loads too I can't imagine how mindblown you will be the day that a rpi has 8x pcie 3.0 lanes available for use 😅

[@CrisanBogdan]

Hi Jeff Quick advice, hack a cheap PSU (the 10 pounds with 450W generic ATX should work), you already have the connectors and always cut the other way of the jacks (e.g the motherboard site) so that you have longer wires to play with!

[@carlsagan2371]

You can buy HP 24 port sas raid controllers for about £40 on eBay. Works out of the box and I have a bunch of drives to hook up to one.

[@ritaoraodndndbe3419]

Amazing work....THX for this....!!!!

[@Yomamura29]

Get this guy a 1M sub please. I am still hoping for a CM4 daughter board that has audio mixer/interface for easy streaming or recording music but still we need more subs! Great vid btw

[@JarrodCoombes]

Pro Tip - Get a paperclip, bend it into a U, then shove it in the ATX motherboard connector and ground out the green (power on) pin, usually right next to it, and you should be good. The paperclip should stay in place just fine and you won't blow up the PSU.

[@grimtagnbag]

Thank u for the boopers and the video. You are awsome love it always

[@betterwithrum]

You know you're kicking ass when you're on a conference call with broadcom engineers for hours...

[@mynameiszoro]

Bro! Just use a (mining) psu turn on extension. There are many verities but personally I use an extension that let you plug a 24 pin connector on one side and a Molex cable from the other side. when the molex is on, it turns on the 24 pin connector psu.

[@SimbaSeven.]

"50 Kernel Recompiles" Gentoo user: Your point?

[@JeffGeerling]

At least you're not an Arch user ;)

[@SimbaSeven.]

@@JeffGeerling Haven't dabbled in Arch yet. My main ones are FreeBSD and Gentoo.

[@axeldz3958]

12:10 looks like you are formatting the whole volume instead of a partition

[@JeffGeerling]

Ah yes. Oops :)

[@bluesquadron593]

This reminded me of your fan install on the dyi rpi case.

[@JeffGeerling]

😳

[@growtopiajaw]

Linus has the Broadcom storage backplane too!

[@JeffGeerling]

I bet RSJ let him 'borrow' it. Those Canadians...

[@recnas]

This is awesome! I'm planing to build an AMD K6-3+ based NAS with hardware RAID. Now I'm really curious to see if it can match the performance and efficiency of the Pi NAS.

[@codewithfrenchy]

YES! we want shirts!

[@Florin-Ciurescu]

I don't know this particular raid controller but older models needed some cooling .It would be great if the sas12Gb run much cooler than the 3/6gb ones .For the PSU you need at least 90w one .Also try pci-e connectors that have separate power plugs , maybe the PCB lanes are not designed to handle that much of power draw and the voltages falls to low .

[@esra_erimez]

Would you please supply a link to the drive enclosure that you are using?

[@JeffGeerling]

The enclosure was provided by Broadcom, but I believe it was originally made by SerialCables.com (since discontinued). I'm pretty sure Broadcom swapped out some internals to get the tri-mode support working, since the original backplane inside was just direct attach SAS.

[@WilliamJasonSherwood]

Obviously we'll need to find cheaper SAS/SATA cards that don't completely over saturate the Pi PCI-e but still have the tools to help the Pi offload as much possible off the CPU. I do love the potential for a SUPER cheap super capable Raspberry Pi CM4 NAS. A SAS/SATA card with a heap of drives plus even USB2 ports for a cheap external USB backup, not to mention the onboard Gigabit Ethernet.

[@danw1955]

I wish I had the resources to put something like this together for long-term NAS storage. However I recently bought an IBM x3650 7945-AC1 2U server for $179, with an 8 bay RAID controlled SAS array, and outfitted it with 8 x 600gb. 10K rpm SAS drives for another $60. With 64gb. of RAM and 2 Xeon 8 core E5620 CPU's, it will handle anything I can throw at it. Only drawback of course, is power consumption. It draws around 140-150 watts at idle and can ramp up to several hundred watts if loaded down. Fortunately, running a Plex Media Server, a couple test web sites and my business software rarely pushes the CPU activity above 10-12%. I guess if I wanted to do a NAS with it also, I could add another SAS backplane and RAID controller and fill the remaining 8 slots with 8 x 1.2TB SAS drives and run XigmaNAS as a virtual machine.😁 (You see what I'm doing here, right? Talking myself into buying more stuff!)🤣

[@liveting4579]

Here's a sub on your way to a mil