New Home Backup Server (Dell T640 with 18 20TB Disks)

[@ScottDotDot]

A couple of things I didn't get to in the video if anyone was curious: Noise was about 55dBA @ ~0.5m from the front of the chassis with the disks under full load (array consistency check). At idle it was 45dBA at the same measurement location. (Though one catch is that it was in a small and otherwise empty/echo-y closet, which probably made it measure louder than it would have been in some kind of proper acoustic chamber. Mainly I was concerned with relative measurements, so it served well for that.) It drew about 105W at idle without any HDDs, and ~235W at idle with all 18 HDDs installed. Under full disk load it was almost bang on 300W most of the time. To put the sound in perspective, the previous setup was about 66dbA. That was mostly due to the HP D2600 DAS that was perched on top of the old server. (I had spec'd the server itself to be super quiet. Didn't measure it independently, but it was much quieter than the Dell for sure.) But one important thing with sound levels: The D2600 fans ran faster and were of a much higher pitch and buzzier than those in the T640. So even with the disks working hard, the T640 "felt" far quieter than just that 11dB difference. It's more like a low "woosh", and not distracting or uncomfortable to be around... though of course that's fully down to personal preference/tolerance. I should be posting a follow-up soon, so you can see and hear what I mean. And here it is now: kzbin.info/www/bejne/iZbGp2iGZZaeaK8

[@gatekeeper88]

Thanks for the vid but please don't gatekeep seller's, it's a really bad look for others who would like to do a project like this or similar, it's really poor form.

[@videofeed99]

You got my subscription after: "... mouth hole on my face..." 17m57s --- Priceless 😂

[@gordoncreAtive]

I think I got the same drives of the same seller - two of them were faulty. One DOA, the other one got bad sectors after my burn in test. I propose you test them. I recorded all the SMART data, put them into a ZFS mirror, filled them to the brink with F3, verified the data, scrubbed the disks and then did a long SMART test. I think that's the most thorough and viable approach.

[@stephenxs8354]

How is this not worse than using a default 4 pass badblocks R/W.

[@gordoncreAtive]

@@stephenxs8354 badblocks can't deal with 18TB drives.

[@ScottDotDot]

Oof, I don't like to hear that. The seller was serverpartdeals. I didn't want to call them out in the video or seem like I was shilling for them, but I guess it's not exactly top secret. Now you're making me doubt myself; I've found that a full array build will usually sus out any immediately bad disks, but half of it was just me not wanting to spend X days running 400TB worth of data through them and then waiting for the array to build afterwards. 😕 FWIW the array has been through about 150TB of writes and a consistency check since the video and so far no [apparent] problems and no SMART issues.

[@stephenxs8354]

@@gordoncreAtive Did you try with block size of 4096 or bigger? My 14tb worked fine with -b 4096.

[@hburke7799]

for the T630/T640 I like using the MB998IP-B (icydock 8 bay 2.5'') and the D P/N 010WY power cable. the 18 bay chassis is a little short on 5.25 space, and comes with a one sata power cable. using the mutli-head power cable saves you needing a power splitter. and the 8-bay icydock has two 8643 mini-SAS connections which are FAR more convenient than using 6 or 8 (especially locking) SATA connectors in such a tight space. the T630 has SAS connectors on the motherboard. though the T640 has oculink, but can be adapted to SAS 8643 fine. (or pcie if you are exceedingly daring)

[@ScottDotDot]

I appreciate the heads up on that. I hadn't really looked through IcyDock's product line, but yeah, a 8643 connectors would be wayy nicer for cable management. Definitely going to keep that in mind for future systems.

[@arbyyyyh]

I’d definitely call those caddies. The things for CD’s back in the day we’d call cassettes.

[@rok9526]

Hi Scott, excellent video, thanks so much for sharing :) Two comments from my side: - Regarding snapshots, you could use btrfs with built-in snapshots (like on Synology). This way you get free data deduplication and still keep the snapshots principle going, to protect against malware. Data would only be "duplicated" in case of being overwritten by malware. - Not sure if it's an overkill, but I do new HDD burn-in with badblocks -wsv You can choose how many runs etc. Takes quite a long time, but gives good confidence imo. My two cents. Thanks again for the great video!

[@Solkre82]

Wooo love tapes. I manage a LTO8 library that just touched 7PB.

[@chrismay2298]

Ah yes, another "mouth hole" user. "Eye hole" is also a favorite of mine. Nice work. Getting a T630 soon.

[@andre-le-bone-aparte]

Question: @09:15 - Do you have a video about your Tape Backup (equipment) setup?

[@MasterJediSean]

I believe that "Sleds" are just a plastic sliding tray with no electrical function like the 12v Hot Swap tray or Caddie. I've heard them referred to as either a Caddie or a SAS Drive Tray. but sleds are just a piece of plastic, which are in most desktop cases.

[@BigBenAdv]

Depending on the system and HBA, some caddies (trays/ sleds) could also incorporate a SAS interposer module. This allows SATA drives to be installed in a system that otherwise would only work with SAS drives (Dell Powervault for example). It basically enables more SAS specific features (e.g. delay spin-up and some SAS specific diagnostics) and allows for dual SAS controller redundancies for the most part. Most Dell PERC cards will also flag a warning if they don't see a Dell FRU firmware on the drive when they don't see T10 (SAS) support on the drives. On more modern servers, they'll offer Value/ Nearline/ Midline SAS which are basically SATA drives with a SAS type controller that doesn't require the use of interposers.

[@primeral]

Just found your vid, this 3.5" x18 is the configuration i want to upgrade my R730 with. It's interesting that the 14g T640 use the 13g drive caddies/sleds. This is great because the 13g caddies are a lot cheaper. I got my R730 idling at 120W with 8x 3.5" drives. I couldn't find it in the comments, have you tested what the T640 idles at with 18 HDDs?

[@TheTrulyInsane]

I usually just batch print all of the labels at the same time, saves all the cutoffs/waste

[@shephusted2714]

connected with gigabit and then you are going to move it? instead run a couple cables and make it 10g bonded but other wise plenty of space, nice setup

[@daven6634]

I'm always curious about what data people are storing on these servers. All I can think of is video formats, like raw editing videos or like movie storage. What else would you do with it?

[@JasonsLabVideos]

Good video sir! The only thing i wish Dell would enable is Bifrication so we can use those multiple m.2 pcie x16 cards..

[@ScottDotDot]

Thanks! Interesting you mention that.. For this project I bought a PCIe card which supports two NVMe drives in sleds that are externally accessible (also from ICY DOCK by coincidence -- ToughArmor MB842MP-B). Was considering using them for my most essential files initially. It said it required bifurcation, so I checked the T640 BIOS and there is an option to turn bifurcation on. That being said, I didn't end up testing it so I can't say 100% it would've worked. But I was kinda surprised the option was even there.

[@JasonsLabVideos]

@@ScottDotDot My Lenovo St550 with Dual 4114's and 256gigs ram stuff with goodies ! Love it but Bifrication isn't an option. You can buy cards that bypass the bifrication but they are 4x the price :(

[@dash8brj]

My R730XD supports this on the x16 slot. I have it populated with a 4 m.2 PCI-E card with 8Tb of NVME drives installed.

[@Yandarval]

The huge empty slot bay cover will be the basis for stuff like card readers to be added with a bezel change. I have an HP server with a similar feature. if you don't need the PCIE lanes or the cores. Taking out the second CPU will reduce your power bill. You are correct about the CPU 2 expansion slots. You need a second CPU to supply the PCIE lanes for them. A similar deal if you want to use all the RAM slots. A second CPU to use the slots for it.

[@tonyc523]

You forgot power draw numbers, with and without drives. Also, dont salute w ur left hand. Nice video. Thanks.

[@FINPJENN]

How many watts does it pull with all drives connected?

[@bak1necWWE]

Hi. why you use hba mode with software raid rather than the raid that was on dell bios?

[@ScottDotDot]

Oh yeah, I should've mentioned that! For the initial array creation, it was so that I could address the disks individually to check SMART stats. But long-term I prefer software RAID for portability. If this system dies and I need to recover the data, I could connect all the disks to any old Linux system. Some older RAID cards and cards from other brands wouldn't necessarily be able to read the metadata on the drives, so I'd need a PERC H740p (or compatible) in the replacement/recovery system. I'd definitely go hardware RAID for performance, but portability is more important to me in this case.

[@Sebyllis7350k]

hardware raid is long dead at this point.

[@RomanShein1978]

Hardware raid is dead.

[@incandescentwithrage]

Lots of comments that hardware RAID is dead. I'd argue hardware RAID is dead for flash. Spinners with the 8GB cache of the H740p is preferable to MD RAID6, IMO. H740p is a rebadged LSI Megaraid and you can import the array with a HBA & mdadm if you ever need to. Smart data and even controller configuration is available from within the OS with perccli Patrol reads and consistently checks can be run on a schedule. Pretty good as HW RAID goes.

[@ChristopherGoggans]

​@@incandescentwithrageI'd say hardware raid is dead even with spinning rust. BTRFS and ZFS offer COW and checksumming, so that your data survives bit-rot and other corruption issues, as well as offering compression and deduplication.

[@jacquesredmond]

What is the power draw at load and idle?

[@jaimeduncan6167]

Many people do not take the rebuild time risk into account. I have explained it many times, and often people are not even paying attention. People nowadays are more and more used to social media information overload and they just feel and react. It's funny because in the hard sciences, engineering and business, we used to look funny to the social science students deciding what is true because it "resonated with them" and today it's mainstream. Flash-based SSDs have a pretty bad write speed (out of cache) but can still perform many more IOPS. They are still expensive, but they already 15TB in the mainstream.

[@lenano]

You might wanna watch out with that sata power splitter for the boot drives, it looks like it's made using those molded sata plugs that are prone to shorting out and catching fire

[@mlitzy]

Have you considered FreeBSD with ZFS?

[@ScottDotDot]

Yup, I use that combo on my main file server. I don't have a 100% solid reason for not using it on this backup system, other than ZFS gets complainy at (iirc) 85% array capacity and I kinda like the idea of using incompatible OSes for data duplication. (Probably pointless, but it's under the vague premise that maybe I'd get a malware infection that targets BSD but not Linux or vice versa... that's crazy though, right?)

[@mlitzy]

@@ScottDotDot I did see on one of the new CanBSD videos it was mentioned that some businesses didn't want a single point of failure so they added BSD systems away from a fully linux environment. So you are in good company going with both.

[@CoreyPL]

Since you are serious about your backups and already have a capable server, then I would definitely split the array at the cost of usable space. Used system, used drives - it's not that hard to have an error on the array and every time it happens it will resilver or scrub. On ZFS maybe 2 RAIDZ2 vdevs or 4 RAIDZ1?

[@jmustafa1973]

Could you do video on the scripts you use to do the backup. I use rsync for remote backup, but nothing fancy like you have.

[@dagamore]

because it is a backup system why did you go with mdam vs zfs, zfs allows for snapshots natively and with scrub/re-silver is an easier process for replacing failing drives.

[@StenIsaksson]

The 6 bay drive cage looks like the one I have. I set the fan speed to low (just to get SOME airflow) and you can't really hear the fans. Very low noise. You can hear them if it set to High

[@Thewickedjon]

what type of business are you in, that you require so much storage / hardware? is this youtube channel THAT BIG?? Ddo you host services for others? or do you just like big expensive server hardware? :D (like me) I'd love to know

[@ScottDotDot]

Well it's not that the channel is big, but the files are big. For some godforsaken reason I record in 2160p ProRes 422 HQ (~220Mbps). Also I may or may not rip my own blurays that I purchased with my own money and duplicated for personal use and archival purposes, and those may or may not be accessible through a Plex server. Plus I'm just generally a data hoarder. Quite a lot of space for full VM disk snapshots of servers as well, which are saved for around 15-30 copies over time. But yeah, most definitely I'm into big expensive server hardware lol. I mean, hyper expensive when new.. not so much by the time I get to it on ebay.

[@TheChadXperience909]

Split it up into 2 x Raid 6 (z2) 8-wide HDD arrays, with a hot spare each. A single one of those arrays should just saturate a 10gbps nic.

[@oldshield]

who is the person on ebay you got the drives thru?

[@dleer_defi]

What a build!

[@rdsii64]

This is going to sound strange, but my back up "server" runs desktop windows. Why? because the back blaze unlimited service is less than $10.00 a month but that service doesn't support server OS's or linux. Right now I have about 50 terabytes of stuff in my media server ( growing very quickly) and its all backed up off sight for less than $10.00 a month.

[@Sebyllis7350k]

if you have thoroughly tested this solution, there's nothing wrong using Windows as a server. updates and other Windows shenanigans should obviously be taken care of, but other than that, Windows is the most popular OS by far after all.

[@KS-wr8ub]

Did you ever try rebuilding from BackBlaze Personal? I've heard very mixed results when you have that much data.

[@ikkuranus]

It might not natively support linux but there is a docker with backblaze personal backup which makes use of WINE and vnc to manage it.

[@rdsii64]

@@KS-wr8ub I have never tried to restore my complete server but I have had to restore single drives and it worked fine. The tier I pay for gives me a year of version history. If I had to restore the complete server I suppose I could do it one drive at a time. It certainly isn't ideal but so far its the only service I can afford with this much data (and growing).

[@debugin1227]

your show so salute, tip your fedora or what ever you like .. just keep up the interesting stuff! and thanks

[@ikkuranus]

How about 4x raid z1 with 2 hot spares? Is it even possible to use 1 or more hot spares to cover multiple vdevs?

[@leexgx]

Your on a good chance that you could have a read error in 2-3 days, z2 minimum Hotspare can cover all Vdevs (z2 before considering Hotspare) in his setup as it's only a backup node, 1 large z3 vdev or 2xz2 Vdevs Also the way he has his backup setup (taring up the changed files) when could just use btrfs on top of md raid and run a snapshot before starting backup pull rsync

[@armstrongskyview2810]

Awesome video, can this host a WordPress site?

[@jeffreylaw4027]

what's the $$ for the build?

[@andyhello23]

You should of had zfs on your backup server, and went with z3, meaning 3disks are parity drives for your backup server. Proxmox could of done this passed through to truenas.

[@BrianThomas]

Dude, im a little new to the channel here. What the heck are you doing that youll need 400 TB of drive space? Man oh man that's a ton of storage.

[@Koop1337]

As soon as you dropped a F bomb I thought "This guy is from the North East" immediately. Instant subscribe.

[@gregsb3454]

Suggestion put the date on the disk

[@punch3n3ergy37]

Nice video! Could you maybe link the ebay reseller?

[@davelb54]

Ditto on this request. A reliable ebay reseller of enterprise large capacity drives is golden!

[@ScottDotDot]

It was serverpartdeals. So far so good with the drives.. I know it took me forever to get to ur comment (sorry about that!) but at least I can now say the 18 in the system have been running OK for over a month now (and the other 2 passed some basic testing). But I don't want to 100% recommend them either, because for all I know the drives could all be duds that crap out after 3 months. But I suppose that's the risk with any refurbs (or new drives come to think of it..)

[@youtubegarbage4u]

what type of raid, filesystem and volume manager you using on top of it? i believe you mentioned ubuntu OS.

[@MrSamadolfo]

😏👍 Very Nice, now fill it up 😊

[@bobkoss280]

New subscriber. Need a homelab tour.

[@kopiluwak_kayubakar]

this not for home, this suited for small business. from my experience this kind of storage server is hot and very noisy almost like enterprise server class. you should has dedicated room with good air circulation and insulation.

[@ScottDotDot]

I just posted a pinned comment with some followup on the power/noise. 100% that server is targeted at the small biz market, but as servers go it's -relatively- quiet and not very power hungry. The noise really comes down to personal preference; Some people could live with it happily, but it would drive others crazy. ~235W power consumption, so not a crazy amount of heat.

[@chenxuewen]

二手还是很划算的

[@billzhou2481]

国内18盘的T630/T640准系统贵的离谱

[@magog6852]

power consumption = terrible.

[@OfficialMikeJ]

The constant swearing when I am trying to teach my kid about computers made me very angry having to tell her to go do something else. She doesn't need to hear this kind of language when being taught about computers and servers. very disappointed you kept throwing out the F bombs. Not acceptable by me. Not impressed at all. no need to be swearing.

[@magog6852]

teach your kid how to be a man, unlike his father