INSANE PetaByte Homelab! (TrueNAS Scale ZFS + 10Gb Networking + 40Gb SMB Fail)

[@HomeSysAdmin]

2:36 Ooooh that perfect drive cube stack!! Wow 1PB in a singe array - you're making me 8x 18TB look tiny.

[@DigitalSpaceport]

I had a hard time committing to taking them and putting them in the jbod the stack looked so good.

[@BigBenAdv]

You probably need to look into NUMA and QPI bus saturation being the issue on your Truenas box since it's and older dual-socket Xeon setup. Odds are the QPI bus is saturated when performing this test. For some context: I've successfully ran single connection sustained transfers up to 93Gbit/s (excluding networking overheads on the link) between on Windows 2012 R2 boxes in a routed network as part of an unpaid POC back in the day (2017). Servers used were dual-socket Xeon E5-2650 v4 (originally) w/ 128GB of RAM, running Starwind RAMdisk (because we couldn't afford NVME VROC for an unpaid POC). Out of the box without any tuning on W2012R2, I could only sustain about 46-50Gbit/s. With tuning on the Windows stack (RSC, RSS, NUMA pinning & processes affinity pinning), that went up to about 70Gbit/s (the QPI bus was the bottleneck here). Eventually, I took out the 2nd socket proc for each server to eliminate QPI bus saturation and the pinning/ affinity issues and obtained 93Gbit/s sustained (on the Arista switches running OSPF for routing, the actual utilization with the networking overheads was about 97Gbit/s). The single 12C/24T Xeon was only about 50% loaded with non-RDMA TCP transfers. The file transfer test was done with a Q1T1 test on Crystaldiskmark (other utilities like diskspd or Windows Explorer copies seem to have some other limitations/ inefficiencies). For the best chance at testing such transfers, I'd say that you should remove one processor from the Dell server running Truenas. 1) Processes running on cores on socket 1 will need to traverse the QPI to reach memory attached to socket 2 (and vice versa). 2) If your NIC and HBA are attached to PCIe lanes on different sockets, that's also traffic that will hit your QPI bus. 3) Processes on socket 1 accessing either the NIC or HBA attached to PCIE on the 2nd socket will also hit your QPI bus. All of these will potentially end up saturating the QPI and 'artificially' limit the performance you could get. By placing all memory, NIC, and HBA to only one socket, you can effectively eliminate QPI link saturation issues.

[@DigitalSpaceport]

Incredible info. Thanks for writing this up I will remove a processor and test this out with the nic and HBA attached

[@rodrimora]

I believe that the windows explorer copy/paste is limited to 1 core go that would be the bottle neck. Also I think at 14:40 you said the "write cache", but the RAM in ZFS is not used for write cache as far as I know, only for read cache.

[@DigitalSpaceport]

Yeah I'm checking into robocopy GUI here. I spent a day trying to get smb multichannel to work with the other 10gb nic in the computer so I hope to be able to track it down soon. In the past my Ryzen 3600 stomped this transfer speed so your spot on. I thought ram buffered writes in ZFS?

[@xxcr4ckzzxx840]

​@@DigitalSpaceport Rodri is right here. Single Core only for Windows explorer copies. SMB Multichannel is a PAIN between Windows and Linux. If you ever get that to work reliably make a dedicated Video about it PLEASE! For the buffered writes; you are right here. ZFS buffers ~5s of writes in RAM and moves it to the Disks. Thats btw the reason Disk benchmarks on ZFS are useless and also why the copying speed flactuates quite a bit on most spinning rust setups. There is an option to tune it ofc, so that you buffer exactly as much data in RAM, as your drives can write until the RAM buffer is full again. If I only could remember the Name... EDIT: If SMB Multichannel is a nono, then try NFS v4 with a Linux System. It will perform substantially better, as SMB is single-threaded only too iirc. EDIT2: openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-txg-timeout - Have a look at that. Its the Cache thing above and might need tuning in your setup. BEWARE! This is a deep, deep rabbit hole!

[@pfeilspitze]

19:38 "now we have this set up in a much more common-sense [...]" -- I'm a ZFS noob, but is 60 disks in a single Z2 really a good idea? Seems like the odds of losing 3/60 disks would be relatively high, particularly if they all come from one batch of returned drives. What if it was 6x (RaidZ2 | 10 wide) instead, say? Then it could stripe the reads and writes over all those vdevs too...

[@CaleMcCollough]

He must be single. There is no way the wife would allow that much server hardware in the house.

[@quochung9999]

He paid off the house I guested.

[@GaryFromIT]

He does in fact have a wife, she even videos him with it for hours at a time.

[@slug.racing]

Maybe you should borrow some big boy pants.

[@verschworer1494]

For example my not wife but girlfriend, in which apartments i have living (God save the tinder)😅, keeping two mini horses, and i just help with them, and i have 30 unit rack where sm blade, 2 switches and kvm, apartments power supply only 6KW, so i think a cant have more🤷‍♂️, but have no restrictions from my gf side

[@RickMyBalls]

@@quochung9999 you... guested??

[@PeterKocic]

Hey just wanted to say your videos inspired me to purchase a DE6600 and they were invaluable to the decision. The result has been so friggin' good. Extremely happy!

[@DigitalSpaceport]

Oh cool glad it helped. These are some massive machines and have a lot of ticky parts, but once running they are great.

[@arigornstrider]

That stack of 20TB drives! 🤤

[@DigitalSpaceport]

#density

[@TheSouthernMale]

He is just trying to insure he keeps a head of me in the pool 🤣🤣🤣🤣

[@DigitalSpaceport]

that word you use there "Trying"....are you sure your using it right?

[@TheSouthernMale]

@@DigitalSpaceport Of course I am, be careful or one day you will fill a strong wind as I pass you by. 😛 Maybe someday you and I will be number 1 and 2 in the pool, you of course being the later. 😛😛

[@TheSouthernMale]

@@DigitalSpaceport You also seam to forget that while you are using compressed plots to stay a head of me, I am not so far, just imagine that strong wind as I pass you by once I compress them. 🤣🌪🌪🌪🌪

[@philippemiller4740]

60 wide raidz2 doesn't make much more sense haha. Try 10 wide raidz2 x 6. That would make much more sense no? Maybe you're limited by smb, have you tried using iscsi or nfs?

[@punx4life85]

Awesome vid! Thanks g! Picked up another 66tb for my farm

[@baystreetwolves]

Love catching up on your build. You never stop building.

[@DigitalSpaceport]

I'm going to get into software more at some point here, but man machines are fun!

[@Mruktz]

I have a humble homelab, but what would you even realistically need a petabyte storage system for?

[@DigitalSpaceport]

video on that all soon

[@RickMyBalls]

@@DigitalSpaceport video full stop

[@chrisumali9841]

thanks for the demo and info, MegaUpload lol... Have a great day

[@LampJustin]

You don't plan on using a single Rz2 in production, right? Right? One Rz2 shouldn't be much wider than 8 drives for optimal performance and redundancy. Recovering from a failure with a 60 drive z2 would take a freaking long time and chances are really high that other drives will go boom as well. It has to read all 1PiB after all...

[@DigitalSpaceport]

Nooooo. I have a second video filmed with the same array and it's much more common sense and safe in it's layout.

[@LampJustin]

@@DigitalSpaceport good good, the other setup would have been pure insanity :D

[@BloodyIron]

Thanks for showing these examples!

[@DigitalSpaceport]

I try man I try. It's a lot of guesswork over here so appreciate the feedback on things you find valuable.

[@BloodyIron]

@@DigitalSpaceport Well I've more been watching a bunch of your eps for IT specific infra, not the crypto stuff myself. So thanks again!

[@andreas4959]

Wicked setup man! I saw you tried running these with the IOM6 without luck, but I have heard a couple of people claiming that IOM12's work in these?.. Do you have any knowledge about that? I'm looking to build a proper storage setup with a DE6600 instead of having multiple old PC's filled to the brim with drives, but really not finding any info on whether or not the IOM12 would work in these...

[@DigitalSpaceport]

I do not think it would work with the IOM12 myself but you should checkout my full video on the DE6600's if you have not yet. kzbin.info/www/bejne/b6jaZ4Gso5WUirc There is also a version that is SAS3 and I can tell you for sure, you want that vs the sas2 version if you are thinking of ever maximizing your HDDs potential fully all at the same time like say ZFS can do. 60 disks at SAS2 speeds are limited to around 1/3 of their max performance. 24 drives is the full utilization for a SAS2 JBOD.

[@caiolopes12]

How did you get TrueNAS running in a infiniband switch? I always hear that IB is not supported by TrueNAS

[@electronicparadiseonline2103]

That's freakin insane. Your out of your mind DS. That's a ton of storage and you look like you just came home from the grocery store or something.

[@contaxHH]

will the floor collapse?

[@DigitalSpaceport]

No but I did static load calculations on the 4" 6 slump reinforced slab in the garage and decided to put plate steel squares under the risers to distribute the load slightly better. So far zero issues. It should be okay even fully loaded, which it's not. Calculated to 800, 1200 and 2000 lbs per rack from left to right facing from front. Good question. Also don't put racks on wooden subflooring if they approach 1000 lbs without additional load deflection bracing.

[@laughingvampire7555]

that sound of fans is just relaxing to me

[@DigitalSpaceport]

10 hour server white noise video hummm yeah

[@guytech7310]

What do people store where you have hundreds of TB of storage? I have at most 40TB (mirrored) of storage, that includes an ESXi, 24 Camera NVR, DLNA, & a few other servers.

[@DigitalSpaceport]

GIS data and project files is what I store as it is useful to my work. Data hoarding newspaper clippings is pretty cool, in a very boomer way, also.

[@guytech7310]

@@DigitalSpaceport Still a Petabyte is a lot of storage. I do manage about a PB of data for a company, but not for a home lab.

[@RickMyBalls]

bluray rips

[@juaorok]

That awesome Right now I have a Supermicro SC836 16 bay with 7 x 12tb hdds and 96Gb of ram I'm upgrading little by little, saving money to upgrade my network

[@DigitalSpaceport]

I'm fairly unhappy with my 40Gb performance but attribute it to the EPYC not having a high core speed + smb multichannel not working as I was hoping. The 10Gb nic is an easy win on almost any machine and fully maxes out here however.

[@christiandu7771]

Thanks for your video, can you tell me where you buy these disk (not available in your shop) ?

[@DigitalSpaceport]

Hard drives do sell fast. We notify paid channel members first of new stock as soon as it is posted to the store. Members also receive a 3% or 5% discount code depending on which level they sign up for (a code is generated for each month which has unlimited use while the code is active.) kzbin.infojoin Another way you can get notified is to subscribe to the e-mail list on shop.digitalspaceport.com. We send out an e-mail notification when hard drives come back in stock if there are any left after the channel members have been notified.

[@samishiikihaku]

Not sure of the differences, but Dell, before EMC, used the same enclosure style as well. PowerVault MD3060E and other varities. Though the prices may be a bit different.

[@DigitalSpaceport]

NetApp made both of these variants and there are not meaningful differences that I have seen at all provided you use the SAS controller for the management node. Just stickers and of course the Dell front bezel. I do have some of the Dell branded EMMs and they work out perfect.

[@samishiikihaku]

@@DigitalSpaceport Yep. Just wanted to show another option. Incase people can't find the netapp version.

[@TannerCDavis]

Arn't you limited to 6gbps sas cable connections? Do you have multi path option on to get above 6? The speeds above 12gbps are probably due to writing to ram, then slows down to write to disk thru the wire connections.

[@DigitalSpaceport]

SAS2 is pretty decent if you have wide mode running. The DE6600 can do that on the first connected device but not on the second daisy chained (that I have been able to figure out at least)

[@thumbnailqualityassurance7853]

How does TrueNAS know how to light the disk failure LED on the netapp if a disk fails?

[@ernestoditerribile]

It’s not TrueNas doing that. It’s the HBA/RAID/Disk controller of the NetApp checking the S.M.A.R.T. Status

[@TheSasquatchjones]

My God! it's been a while. Great video!

[@DigitalSpaceport]

Howdy 🤠

[@TVJAY]

I am new to your channel, is there any chance you can do an overall tour of your setup and how you got to where you are?

[@DigitalSpaceport]

What a great idea. I'll get one in the works here. Welcome and thanks!

[@何旦聃-c7u]

How do you solve the problem that the indicator light of the sata hard disk is not on?

[@DigitalSpaceport]

The SATA light is technically on, its just very very faint. It does not show well on the JBOD with the camera. If it is important to you to have the full blinking light, then an SATA to SAS interposer would be needed.

[@ewenchan1239]

I couldn't find the comment where you asked me about SMB Direct in Windows 11, but it does look like that if you go to Control Panel -> Programs and Features -> Turn Windows Features On and Off -> SMB Direct -- it does appear that in Windows 11, you can enable it. Just thought that I would pass that along to you.

[@notmyname1486]

just found this channel, but what is your use case for all of this

[@mitchell1466]

Hi, loves your video I noticed when you where in the iDrac you were on a Dell 720XD, I am looking at going to 10GB for my setup and was wondering what 10GB NIC you have installed?

[@DigitalSpaceport]

Mellanox connect x2

[@mitchell1466]

@@DigitalSpaceport hey thanks for the reply Did you have any difficulty getting it to work with Scale or did you just plug it in and TrueNAS picked it up?

[@JasonsLabVideos]

Nice setup !! Looking skookum man !! Keep going !

[@DigitalSpaceport]

Thanks appreciate it 👍

[@xtlmeth]

What SAS card and cables did you use to connect the JBOD to the server?

[@DigitalSpaceport]

LSI 9207-8e and 8088-8088 cables. Linked in description to exact ones I bought.

[@apdewis]

The power bill on that setup must me monstrous. My more modest setup of R630s and 320s costs me enough. I am envious.

[@DigitalSpaceport]

One would think but I'm base rate 10c/kWh and so it isn't that bad. I also shed load dynamically via HA and proxmox and don't run all the machines at once very often, unless needed. Usually about 250 total with the house for the electric bill. Cooling is consistently the largest user in the garage.

[@apdewis]

I can only envy that per/kWh rate as well. Mine is somewhere around A$0.22. That said still ends up being a lot better value than AWS is at work...

[@StenIsaksson]

I heard something about Windows not being supported with Epyc CPU's Wrong apparently

[@DigitalSpaceport]

Yeah so just use the ryzenmaster drivers and it works great. I was discouraged initially also by what I had been seeing others say.

[@visheshgupta9100]

I was wondering if you could do a video on TrueNAS scale with multiple nodes. There is no video on youtube that discusses this in detail. In layman terms, I would like to deploy 3 different servers, and control them all from one place. My question is do we need to install TrueNAS scale on every server? Or is it that we have 1 TrueNAS Scale server, and others are TrueNAS Core?

[@DigitalSpaceport]

If you need to use True Commander (the software that manages multiple nodes) I think you need to contact them about a license. I have heard its affordable by I don't really know that. You dont mix and match Core + Scale really.

[@visheshgupta9100]

@@DigitalSpaceport Thank you, that is exactly what I was looking for.

[@visheshgupta9100]

​@@DigitalSpaceport I am planning on building a homelab, I was thinking of having multiple servers for different kinds of media. For instance, one for game storage, the other for critical data & backup, and so on and so forth. So essentially, I wont need all the NAS running at all times. I want to be able to power on just the system I need, so it would work as a standalone system, and also the ability to control all the systems from one place so that I don't have to configure users/permissions/shares for each and every system individually.

[@visheshgupta9100]

​@@DigitalSpaceport I was considering TrueNAS at first, but now I am kind of leaning towards UnRaid due to their recent implementation of ZFS. So essentially I could use UnRaid known for its parity, or use ZFS known for its speed and reliability or use both at the same time. In terms of flexibility, to mix and match drives, ease of use, low hardware requirements, I believe UnRaid has an upper hand. What are your thoughts?

[@MHM4V3R1CK]

@@visheshgupta9100 I think truenas command is free now. Truenas scale is the Linux/debian flavor of truenas and core is the older but very stable BSD flavor btw

[@linmal2242]

Does this array run 24/7 or is it powered down most of the time? What is your power bill like?

[@DigitalSpaceport]

Runs 24/7 and is around 2.2 amps at 245V so 540w per jbod. Per disk that is 9 watts which is among the best per tray efficient jbods I have measured

[@maximloginov]

Hello. What kind of raid you finaly using for plotting? Stripe? Raidz2?

[@DigitalSpaceport]

RAID0 of 12 disks. So far none have blown out and the performance is awesome. Worst case one blows out and I need to replot it, not a huge deal. I can do that in a few days.

[@snapu-g1j]

Quick question, could the DE6600 handle SATA HDD or only SAS? Could i just buy a SAS HBA card and plug it into my ubuntu server? Thanks

[@DigitalSpaceport]

It handles SAS or SATA. Get like a 9207-8e vs say a 9205 or 9200

[@snapu-g1j]

@DigitalSpaceport thanks for the reply. One controller can handle all the 60 drives with a acceptable response time? ( chia farming ;-) )

[@何旦聃-c7u]

Hello,Is it possible to update the ESM/EMM firmware?

[@TheDropForged]

Serious question, I see people with server equivalent to enterprise. Do people really need that size?

[@TheDropForged]

Aa you do crypto stuff

[@DigitalSpaceport]

Yeah ai had a smaller half rack prior which is more than enough nowadays for most homelabbers

[@capnrob97]

For a home lab, how could you even begin to use a petabyte worth of storage?

[@jonathan.sullivan]

I'm interested in seeing what you can get for performance with multiple Vdevs, Tom Lawrence has a good breakdown video for logically how many disks per vdev one hsould have for performance. 1 Raidz2 vdev across 60 disks def isn't it but it's fun to see. #subcribed

[@DigitalSpaceport]

For sure it was just to show the size lol but I am working on a followup here. Looks like I have a solution for the pathetic SMB transfer speeds also.

[@watb8689]

you have some insane homelab. how is the energy bill coming

[@DigitalSpaceport]

runs 225-250 per month. Not really that bad.

[@fisherbu]

nice job! how to make a plot only 74gb?

[@DigitalSpaceport]

Gotta create it with the Gigahorse or bladebit cuda. Bladebit isn't farmable yet, but Gigahorse is

[@hescominsoon]

try it over iscsi isntead of smb. also sclae does not perform as wlel as core does across the board in my own testing. unless you want to run vm's then sclae is the way to go. if you want only storage then core is your best bet.

[@DigitalSpaceport]

I've had a lot of folks tell me to go core for performance and I'm going to check it out. I use proxmox for VMs also so no reason really. I will do a video on iSCSI I think after I learn some more about it and give a "noobs perspective" to iSCSI. I used it only once in the past via a Dell MD1000i and it was a painful thing as a result of that. Time for another round!

[@hescominsoon]

@@DigitalSpaceport iscsi in core is easy.... I do not use proxmox as my hypervisor so I don't know about setting up from iscsi on that end.

[@rfitzgerald2004]

On your shop, do you ship to UK?

[@DigitalSpaceport]

Yes just make sure to add your phone number on the form. Customs will stop the shipment w/o that.

[@trousersnake1486]

This is waaaay above my knowledge base of pc hardware but its impressive what I do understand. Looking to upgrade from my ryzen 5900x x570 system to an eypc system when finances allow.

[@DigitalSpaceport]

Its funny how things get out of control in life lol

[@trininox]

How's the electric bill?

[@DigitalSpaceport]

250/mo

[@ewenchan1239]

The other problem that you might be running into it looks like according to wiki, your Intel Xeon E5-2667 v4 are only 8-core/16-thread processors each, which means that having to deal with/manage a 60-wide raidz2 ZFS pool is going to tax the processor quite heavily when it is trying to manage that many drives with only 16-cores/32-threads total (possibly). Keep an eye on out on your %iowait.

[@DigitalSpaceport]

I have part 2 filmed here of me trying out other more logical configurations then 60 wide. Out this week!

[@ewenchan1239]

@@DigitalSpaceport I look forward to watching that video. (The reason why I mention to keep an eye out on your %iowait is because my server has 36 drives, 32 of which are handled by ZFS under Proxmox, and under heavy disk load tasks, the %iowait will jump/start to climb until those tasks are finished, and then the %iowait will fall back down.)

[@MyersJ2Original]

Are those drives used you sell or new?

[@DigitalSpaceport]

Refurbished from Seagate but they dont have power on hours like used pulls do.

[@marcelovictor3031]

how many hds of 20 tb do you have?

[@DigitalSpaceport]

The real question is how many of 22TB do I have😜

[@skyhawk21]

Need help, got whs server with 50tb of drives, need a cheap good quality 2.5gb switch maybe with 10gb port and also cheap quality 10 gb card for server??? 1gb don’t cut it

[@DigitalSpaceport]

Skip the 2.5G and just roll out 10Gbit. You will be much happier. This mikrotik switch is great to start with. geni.us/goNi9C

[@bokami3445]

OMG! How long would a scrub take on this monster!

[@carbongrip2108]

I hope you enabled Jumbo Frames on all your NICs...

[@DigitalSpaceport]

Yes I do. I have a home side segment that bridges to these devices but the big gear is all on Jumbo

[@Murr808]

What do you do about windows updates ?

[@DigitalSpaceport]

Suffer through them eventually. They only allow you to postpone so much unfortunately. I try to do them when behind KB

[@jondonnelly3]

You can block it by added some entries to the hosts file.

[@gustcol1]

I have the same problem with my 100Gbps network..

[@ewenchan1239]

Unless you're writing to an array of enterprise grade NVMe U.2 SSDs, 100 Gbps for storage for a home lab user -- you'll never be able to hit more than a few percent of the 100 Gbps line speed/capacity. (I have 100 Gbps as well (Infiniband).) Even if you enable NFSoRDMA, if you're going to be using spinning rust, it's not going to make THAT much of a difference. (The highest I've been able to momentarily get is about 32 Gbps kernel/cached writes. More often than not, my system hovers around 16 Gbps nominal max.)

[@Firesealb99]

You had me at "caddies not needed"

[@DigitalSpaceport]

It feels good to not need to screw in caddies for sure

[@Solkre82]

Don't want to brag, but I have mirrored 18TB drives in my NAS and I almost have 6TB of data on it. 💪💪💪💪

[@DigitalSpaceport]

Bragging about storage is allowed and encouraged here

[@seelook1]

I'm going to sound like a kid. THIS IS SO COOL! I WANT IT. My wife will kill me if i ever add something like this lol.😅

[@DigitalSpaceport]

You have 1 life. Live your best one. (advice that gets folks divorced hahaha)

[@blueprint4221]

watch that bend radius, sir

[@DigitalSpaceport]

so about that.... Needed this comment before the video lol

[@GreatVomitto]

You know that your raid is fast when you are limited by the CPU.

[@DigitalSpaceport]

A never ending problem except I got some RDMA going now!

[@DennisJlg-cu1vw]

Hello my tip do not use Win 10 or 11 but a Win server why, Alex from The Geek Freak found out that there is a limit to the network in normal systems and this is not active in servers. For info

[@DigitalSpaceport]

This is easy to test out! Will toss a server 2022 on and see what kinda perf benefits I can get

[@FakeName39]

This dude running amazon from his home

[@TheSouthernMale]

Hey, if you see me leave the Pool its because I am losing money on it. I calculated that if I was solo farming I would have earned 4.5 additional XCH in the last 10 days. I will wait and see if the wining trend continues.

[@DigitalSpaceport]

I have gone that path indeed sir. GL to you, if you show back up I'll know why 😂

[@TheSouthernMale]

@@DigitalSpaceport I am not saying I will leave yet, but if I do leave and then come back it will only be to jump a head of you in the pool. 😛

[@trillioncap]

wow invredible

[@jondonnelly3]

Why do you need a PetaByte as a home user/ small office user ?

[@DigitalSpaceport]

I have a lot of isos

[@rsqq8]

This man ISO’s 😂

[@Alan.livingston]

That’s a lot of adult material you seem to be hoarding there, sir.

[@DigitalSpaceport]

The backups backup has backups

[@MelroyvandenBerg]

Cat door. You get it? 😅

[@rustyZ_stacks]

5 axis stabilization makin me dizzy ahaha

[@DigitalSpaceport]

Ill try to use less 🚁

[@nexovec]

Uhh 60 drives in one vdev :D this video is so wrong :D Good job.

[@franciscooteiza]

Not if you are using dRaid (not the case in this video)

[@sm8081]

Envy….Bad feeling, I know…😅

[@pudseugenio4118]

looks like a gold bar from a cave

[@DigitalSpaceport]

YARRRR 🏴‍☠️🦜

[@NilvinPerpinosas]

that is not a homelab, that's corporation shit bro

[@Paberu85]

I wonder why would somebody do such a thing to his own house, and wallet?..

[@ajandruzzi]

I assume you’re looking for an answer better than “because he can”.

[@DigitalSpaceport]

you and me both

[@charliebrown1947]

you need higher MTU. you're breaking these files into too many 1500 byte packets (think DDoS attack). btw i hope you didnt keep that 60 wide raidz2 configuration... that's silly. Oh, you also have your ARC set to use 50% of your ram (default) and half your ram is literally doing absolutely nothing at all. change the default to allow it to use 90% or even more

[@leo_craft1]

They cost 1000€+

[@InSaiyan-Shinobi]

Is this at your house Wtf?

[@NilvinPerpinosas]

143

[@jj-icejoe6642]

All that wasted money…

[@16VScirockstar]

The throughput is constrained through the PCIe 3.x bottleneck, de.wikipedia.org/wiki/PCI_Express

[@DigitalSpaceport]

I think SAS2 is bottlenecking even before PCIe3 here. I have another video in the works that will check into but pushing the limits of my spindles is a topic I am diving more and more into.

[@16VScirockstar]

@@DigitalSpaceport nice! Just out of curiosity, what do you need this setup for? It must be immensely expensive. As a former SAN admin, I can tell, striping over more disks doesn't give you more linear performance. I remember the threshold, 17 years ago, was at around 10 disks per stripeset.

[@FunkyKong]

@@DigitalSpaceport How is it possible that 8 lanes of SAS2 (48Gbps / 6GiB/sec) are saturating when your throughput is < 2GiB/sec? In my experience pulling my hair out doing exactly what you're doing in the 2nd half of your video, Windows just seems to not manage above 10-15Gbps no matter what you do, even using network testing tools. I am using 100G Mellanox ConnectX-5 cards with exactly the same limitations and no amount of driver tuning seems to get there. But if I reboot into Linux I can hit wirespeed between machines. The only solution for good high-speed SMB on a Windows client, particularly for single streams (where multichannel cannot help) is with RDMA, aka SMB Direct. That requires Windows Server on the other end, or ksmbd if you dare go down that rabbit hole. I hope for an update with whatever you've found since this video.