Upgrading our TrueNAS File Server to 300TB - Supermicro & Seagate

[@danielfisher1515]

Thanks for sharing your upgrade process. You got 12x16TB from Seagate, with 185 subscribers? Baller AF!

[@Bytesector]

Hey Daniel. Not my first rodeo thankfully and once in a while we hit a jackpot (snagged an EVGA 3070 back in late 2020 as well). Bytesector ran as a web publication from 2001-2008 and became quite a large presence in the review community. We're working to rebuild our presence and influence slowly but surely. I hope you subscribe and stick around.

[@Bytesector]

Glad you liked the upgrade process. We're trying to build out our content around the server environment in a home setting. My latest pet project I'm trying to get off the ground is implementing a blade server to replace my current ESXi cluster.

[@iXsystemsInc]

Fantastic video! Go TrueNAS!

[@Bytesector]

Loving it so far - 5 years into my usage of the TrueNAS platform. Aiming for a petabyte in the next 2 years!

[@georgiaguardian4696]

Greta pace to explain and proceed with each step. Just subscribed! Your channel is more useful for me than some big name youtubers by a lot more!

[@Bytesector]

Thanks for the vote of confidence George! I really appreciate it. I'm just getting back into producing more content. I hope you stick around for the long haul.

[@crzymac]

Can you tell me the model/part number for the rails they sent you? I'm looking for a set for my chassis currently. Thanks!

[@Bytesector]

I'll have to check and get back to you. It requires accessing the side of one of the rails in the middle - not the most convenient place to find a model number but i'll do the work and get back to you.

[@themikemac]

@@Bytesector That would be great! Much appreciated!

[@jmpalacios]

Awesome video! Could you please provide some info on that cable management arm that you mention? And any other cable management tips you might have? I'm lacking good cable management in my rack because of overly long CAT6A cables I originally bought, and I'm looking to improve that, thank you!

[@Bytesector]

Most servers have optional cable management arms that can be attached to the back of the rails and bend so the cables remain organized when you slide the server out for maintenance. You can see an example of the cabling job I did in my server rack back in January here.. Before: twitter.com/bytesector/status/1352047973639843840 After: twitter.com/bytesector/status/1352310611460444164 This is a problem a lot of people struggle with, myself included. Another option is to grab some velcro straps and cut them to the length you need and start bundling cables. I hope that helps. Please don't forget to subscribe and help support us and our content. We appreciate everyone who joins the community and look forward to having you around. Thanks.

[@jmpalacios]

@@Bytesector Thank you for the hint and info, appreciated!

[@frederichardy8844]

Dah yes it' heavy. It's why you put the drive in after installing the enclosure... Another reason is if you drop the enclosure (always a risk, ask LTT Linus...) during the installation at least you don't damage the drives.

[@Bytesector]

Fair except 1. I'm not Linus...hehe and 2. The midplane drives couldn't be added later due to the location of my enclosure - beside a wall and without enough front clearance the pull the chassis out fully. As a result I cant insert or remove screws holding the top of the chassis on while it's in the rack. As for the front drives, yeah, I could add those post install.

[@nelenele246]

What is the model and configuration of the Supermicro server? What is the model of the HBA with internal SAS connection?

[@Bytesector]

Hi, this is the model: www.supermicro.com/en/products/chassis/2u/826/sc826se1c-r1k02jbod

[@JoanNin]

GREAT! Love TRUENAS. Question for the new server micro you need to install TrueNas + the external SAS cable? or just the SAS card plus the cables

[@Bytesector]

Well the SuperMicro chassis doesn't have storage, it's just an external hard drive enclosure. TrueNAS goes on your server and the SAS cable connects up. The drive enclosure should be powered on before the server.

[@chrismoore9997]

With only 300 subscribers, how did you manage to get these vendors to give you hardware?

[@cdtluna]

LOL just BS...he must have had to pay for them or got a small discount...if he did get them free then I dont have any respect....I should just open up my own channel and get free stuff then....lol

[@chrismoore9997]

You would normally put the chassis in the rack first, then install the drives, because of the weight. I have server chassis where I work that have 60 drives in them. Add the weight of the drives to the chassis and you can't move it without a hydraulic lift, which I happen to have but it only goes up to half the height of the rack.

[@Bytesector]

I completely agree and normally I would do that, but middle section that lifts up has a couple screws on the side of the chassis that hold it securely in place and since I wasn't using rails at the time, I couldn't secure that panel after rack installation - the drives had to be installed first.

[@majstealth]

now imagine a 90 3,5" drives system...^^ i want one, but surely cant afford the chassis, nor the drives, nor the power draw^^

[@JzJad]

I'm curious if you have had a drive failure and what the rebuild time was, 16TB per disk in that small of a build is kinda scary

[@Bytesector]

Good question and I don't really have a great response unfortunately. I've only had one failure and my hot spare filled in before I knew it had happened. So let's say less than 12 hours as it was an overnight failure discovered in the mid morning. Having so much data per disk is scary but then that's the huge benefit of ZFS. You sacrifice a lot of space for parity but it pays for itself in spades when there's a disk failure.

[@JzJad]

@@Bytesector oh I'm aware I just have not seen alot of systems with over 8-12TB disks yet, I personally have 5 x raid z1 | 25 wide, so 25' 6TB , (3 hot spares) glad to know it took less then a day depending on pool usage that could be great

[@Bytesector]

Well I'm in well over my comfort level - sitting around 89% consumed on the pool. I need to acquire another 4 disks and am trying to decide if i add a vdev with 18tb drives or 20tb drives (and of course the associated hot spare drive). either way, a brutal upgrade on the wallet. My last scrub took 22.25 hours to complete when the pool was sitting around 87% full...to give you an idea of performance with this amount of data (that's across 12 drives sorted into 3x 4-drive RaidZ1 vdevs).

[@ochristi]

@@JzJad Got 3 vdevs with 7 18TB drives in z2, we only had to resilver once, I forgot how long it took. Scrub takes 34 hours. 😅

[@JzJad]

@@ochristi oof, my pool of 72TB, 35TB used only takes 7 hours

[@mauritsl84]

Nice video, how fast can you read/write over Smb and iscsi with 10gbe?

[@Bytesector]

I'm only R/W over SMB and I get nearly a full 10Gbe. I write to the array at about 1.2GB/s and reads about the same. I'm super happy with it but find myself pining for a 25 or 40gbps backbone in the house now.

[@andymok7945]

Nice video. Yeah, supermicro makes great stuff. I got one of their 3U 16bay chassis to build a nice TrueNAS setup. I need lots of storage and am waiting 6 month now on getting my Seagate EXOS 16 HDDs. For now, I can only monkey with small 3 & 4TB older drives to learn TrueNAS

[@Bytesector]

TrueNAS is just one part of it. I actually learned all about ZFS by using Solaris before moving into TrueNAS. There's a fair bit to understand and while the naming in TrueNAS varies slightly, it's all the same stuff. Truly, ZFS is my choice file system due to its resiliency.

[@frederichardy8844]

There's not enought information about the supermicro enclosure but it's possible you should try to put 6 drives on the front row and the another 6 drives in the second row, positionned so they are only next to empty slots. It would be better for thermals and temp is important for HDD lifetime.

[@Bytesector]

I totally agree that from a thermals perspective that would be better. The thing with this chassis though is density and I wanted to fill every drive bay for max density in a 2U form factor. Since publishing this video, I have had a total of 1 drive cause issues and need replacing and that was within the first year. Since then it's been solid and reliable. No complaints on the chassis at all. It's a well thought out platform from Supermicro.

[@sebastianasebastian373]

One thing I found, you didn't plug both power supplies in on that server that you installed the HBA on, if one fails the server turns off, if both are plugged in it can continue running with no disruptions. Also, just wanted to point out that you got everything for free.

[@Bytesector]

100% agree that I should have both PSU's connected but I didn't do it for two reasons. First, I'm trying to save every possible watt and having the 2nd connected still draws power. Second, I only have one breaker strung to the rack so having both only serves to save me from PSU failure not power redundancy. Over the course of years, I can deal with a PSU failure should one occur.

[@fmj_556]

What are you going to use for backup?

[@Bytesector]

I have a 32TB cold storage server that I use for now but obviously, expansion will be necessary down the line. I compress and store on that box. I've opted for ZFS as my file system due to the high resiliency of it.

[@StephenMoreira]

When yea opened that Chassis, I was like 'oh fuck', never seen a design like that one.

[@Bytesector]

Yeah, it's an awesome box. A bit loud on the fans but awesome no less.

[@oldshield]

do you have the model of that chassis and the model of the hba you used?

[@Bytesector]

Hey, it's the SuperChassis 826SE1C-R1K02JBOD. The HBA is a Broadcom SAS 9300-8e. I hope that helps you. Don't forget to subscribe and stick around. We've got some new content coming soon. Thanks.

[@oldshield]

@@Bytesector thank you

[@PiKniOb4]

Awesome build! Would be great if you do a follow-up video on your vdev configuration.

[@Bytesector]

Great idea. I'll put something together. Likely a shorter episode. Anything specific you'd like to see covered?

[@PiKniOb4]

@@Bytesector awesome! I am most interested in design decisions and what other configurations you considered and contrast with the existing vdevs. Thanks!

[@Bytesector]

Sounds great. I'll try and put something together on that.

[@KannonRomano]

whats the model of the supermicro chassis

[@Bytesector]

Hi @KannonRomano, I believe it is the SuperChassis 826SE1C-R1K02JBOD (www.supermicro.com/en/products/chassis/2u/826/sc826se1c-r1k02jbod). Planning a build of your own?

[@CliffR]

what model chassis?

[@Bytesector]

It's the SuperChassis 826SE1C-R1K02JBOD. Planning on getting one?

[@svenstubes]

What Chassis is this?

[@Bytesector]

Hey @Stephen, I've mentioned it in other comments on the video. It's the SuperChassis 826SE1C-R1K02JBOD.

[@UsmanMalik-nv5dz]

What's the model of that supermicro?

[@Bytesector]

Hey, it's the SuperChassis 826SE1C-R1K02JBOD. Planning an upgrade of your own?

[@jonathan.sullivan]

"I'm going to install them in the front 12 bays..." Proceeds to install 12 drives in the midplane.

[@Bytesector]

Good catch. Script wasn't updated and the editing failed there.

[@ZztufNow]

4000 for the drives, 2000 for the chassis. Prices are also enterprise level.

[@Bytesector]

Yup. But the enterprise drives are still cheaper than the prosumer drives for NAS. And yeah, the chassis is not cheap but certainly worth the investment.

[@Botio]

@Seagate Technology I run out my Storage Space as well~~~

[@Platoface]

Yes, we understand low disk space. 😂

[@Bytesector]

Don't we all. I'm in need of a storage expansion already.

[@JasonsLabVideos]

Good video ! Subb'd..

[@Bytesector]

Thanks for the feedback and the sub! Glad to have you around.

[@JasonsLabVideos]

@@Bytesector Any time sir !! Was good content !

[@АлександрФиногенов-к5ф]

😎👍

[@Bytesector]

Thanks for the support. I look forward to bringing more content your way soon.

[@romangsh]

carefully look at the warranty. as seagate drives will fail

[@Bytesector]

All drives fail at some point. Some drives are more prone to failure. There was a time I wouldn't have trusted a WD drive. Maxtor was my old faithful brand. Seagate was rock solid for me until their 4TB drive which failed me left, right and center for a time. Since though I pay attention to drive failure reports and plan accordingly.

[@bentheguru4986]

If I can make some suggestions, understand audio levels on your recordings and post, intro music blown out with distortion from being over-driven and you audio in the video is set a little too high, picking up you voice loud with dynamics and white noise. The other area, less drama, plenty of others doing BS drama. Just talk normally like you would to a person standing infront of you. The pausing and jerking of speech is off-putting. You do have moments were you are natural, just relax and go for it. Dual PSU's in servers are designed to both get power. You only had power in one thus the fault lights. If you are cheap and only want to run one PSU for some reason, unplug the other PSU and slide it back an inch so it keeps rear hole blocked.

[@Bytesector]

Hi Ben, thank you for the constructive criticism. I appreciate it when people take the time to provide valuable feedback. I will take it to heart and just talk naturally - it's one of those things that's easier said than done. As for the PSU's, I'm well aware of the reasons for the fault lights and yes, the 2nd PSU draws some charge and given the current rack draws something near 24kwh/day I'm trying to keep the power draw as low as possible. In fact, I'm desperately trying to figure out a way to subsidize my power supply with something that isn't the power grid - an alternate source and am trying to figure out the best way to do that so I can reduce the cost of running the servers...a pet project of mine that when I figure it out, I will cover it on the channel here.

[@bentheguru4986]

@@Bytesector LOL, you will get hang of the sitting infront of the camera. You already are doing it if you look back through your video. As for juice, yes, I run a full home server rack and know the pain of power draw. Running both PSU's only adds a couple of percent extra draw in some cases, SFA when configured. You can set the PSU's to do what you are doing now but have them both connected to maintain redundancy. Yes, same here, solar is on the cards after we move house and I can setup a decent server room. Keep it up buddy...

[@ginoPD]

Be aware that zfs wants you to have 20% free with a minimum of 10% for a stable runnning system. Having 8% free scares me..

[@drpepper3726]

2.5 million hours is over 280 years.

[@rezenclowd3]

MFGs should have never used "drive trays". They need to be toolless. Same problem with EDSFF...so many wasted man hours installing thousands of drives.

[@Bytesector]

Ubiquiti's drive trays are pretty good and toolless. But I get why they use screws and such - it's just a legacy way of thinking.