Level1: We bootstrapped our own ZFS storage server: 172tb, extremely low cost

[@spacejaga]

god those mini-movies are just on the edge of being most cringeworthy things ever and yet somehow they manage to be funny...

[@Teknickel_ftw]

still better than cable tv

[@thingyee1118]

I love that SOS with your eye lids.

[@pedanticat5830]

It wouldn't be the first time he's done that, lol

[@00101001000000110011]

that really hit the homerun xD

[@YanDoroshenko]

That horror sequence was MAGNIFICENT.

[@imreloadin]

I can honestly say this is the one tech channel that I'm actually excited about when I see it pop up in my subscription box! Keep up the great work guys!

[@SirCrest]

Now THIS is the kind of content I've been hoping for from you guys. Can't wait for more server stuff like this.

[@SgtSayWhat]

I understood about 7% of this, still watched it all though

[@thetruthhurts2352]

NO

[@GNARGNARHEAD]

LOL!.. whoa, those heatsinks... it's amazing how far Krista has come from being on the news, a little socialization can work wonders

[@scruball1375]

That wall had like whole of my KZbin subscription box.

[@martyhastings9347]

i am giving a lot of thought to building something like that for my family (maybe not so big). recently my hard drive failure was a failure that i have heard about but have never seen. a platter broke apart. not sure if it was caused by a head crash or not. Anyway.............after i told my wife we lost lots of pics .......lets just say i think my wife and Krista must have grown up together. i will be watching this video a few more times and looking forward to more of the same type in the future. thanks guys.

[@rogerg0834]

I have to admit..when he said having a barebone Windows image on the network ready to install...that REALLY peaked my interest. We have it at work and I must say it is really cool to do. Just boot off the network and bam..starts the install...major nerd coolness!!

[@MrBiky]

Linus - Super popular, but Canadian. Barnacles - no notes. I don't know the top guy. Paul - Real chill, has cute dogs. Qain - Friend of Wendell, actually ??? elite. Kyle - A true BitWit Jayz - no notes. Jennifer - No girls on the Internet. Ryan = clone. Holy smoke, that's some creepy, but true details.

[@AKAMustang]

"Top Guy" is TimmyTechTV Qain's note says "Actually enjoys Elite", presumably referring to Elite Dangerous

[@scz0]

whose jennifer?

[@daviddupoise6443]

This. This right here. All day everyday. I love this well thoughtout content. It's not the sexiest most expensive hardware but instead it is the smart solution.

[@tkojabroni2833]

Y'alls acting is up there with the likes of William Shatner and Ben Heckendorn

[@andreijecu765]

The SOS in morse code Wendell is doing at the end of the video got me good,killer Krista strikes again. Excellent video,the way you combine useful knowledge with bits is amazing. Than you for this!!!!!!!!!

[@randomtvninja]

"if you can stick with me" are you kidding I've been waiting for heavy videos like this for a long time! I want full hour videos about all of this I need it!!!

[@ibonitog]

Sometimes I come back to this video just for the sketches - and it's worth it every time!

[@wilsonthesphere]

Loved this more technical video I didn't except variety so soon. Also the skit was great and made the pace of this 20 min video seem like is was a 5 minute video. You guys are doing GREAT!!! Keep them coming.

[@MaelosV]

Thank you for this. I'm getting back into my own lab and this is helping me get back off the ground. I look forward to seeing more of your work and steering people your way.

[@EposVox]

Good shit, guys. I wish I had a full server rack to work on. Good deals on bays right now.

[@grassyclimer6853]

the file transfer is coming from inside the house!!

[@CriticoolHit]

::heavy data transfer sounds::

[@chrisbay8578]

I do love it when TechTubers do overkill builds just for the heck of it :D Next time, i dare you to do a 20 GPU server, just to show Linus and his measly 10 GPU rendering machine who's the ballers now :D

[@Level1Techs]

thats the great thing, we spent practically nothing on this and it is still overkill! And, obviously, not sponsored at all. ;) It is GREAT, and made from trash that we have show how/where we get it in other videos. In total we spent only a few thousand dollars on this for everything.

[@chrisbay8578]

" we spent practically nothing.... .... In total we spent only a few thousand dollars on this for everything." I think you and i have different opinions how much "practically nothing" is :P

[@brixhodl4765]

You can buy a 01-02 V12 Mercedes CL600 for around $10,000. That is less than %10 of what it retailed for. Do I think $10,000 is a small amount of money in and of itself.... No. That said, If I was able to acquire a functional used luxury sports car for 90% off, I would still consider that practically nothing given the context. You could build a 172TB ZFS for kicks, but chances are there is a necessity for it, so like Wendell said, I took this video as kind of a hypothetical system architect type build. You could play with file systems/RAID arrays/bootstrapping,etc with much less storage.

[@TheDiamondGames]

It actually is practically nothing for 170TB of storage. You would spend around $6000 for the desktop class hard drives if you put them in a raid 0 array. One shelf would be overkill for many. Even 6 hard drives can be enough for the majority. After all, you should think what do you need this thing for and how much space you need.

[@ChrisD4335]

I wish I was at a point in my life I could say oh this is made of trash, it only cost thousands of dollars and mean it seriously. #lifegoals

[@Banner1986]

I'd forgotten how long ago the LSI 8880 EM2 came out... Your video just reminded me of the days when the 1078 ROC controller was taking the storage world by storm right as I was coming into the storage world, working with the field engineering team over there - not to mention the 6Gb first release, where you had everything from ZFS developers at sun working with 600 disks attached, down to the guy in his basement trying to get a new speed record with OCZs drives lol Thanks for the blast from the past Wendell!!

[@ScullyBrewing]

These are the types of videos I love, as a systems admin student in college, this is my jam

[@johanneskaramossov5103]

awesome to see this, I only have some hobby projects myself using old enterprise hardware that i got cheap.. actually for my own experiments considering enterprises tend to dispose of them instead reselling them getting enough storage at a decent prize is my main issue(closely followed by finding the right finstacks/coolers for the cpu's in 2U and 1U racks that i can buy used).. really helps me understand the concept without needing to get all the stuff myself

[@tachmonite37]

Level 1 shows are just awesome. I love their humor and dialogue...and the information of course. great job guys. i cant get enough...

[@YumzHD]

Loved the episode Wendell, keep it up the channel has been amazing since launch. Best of luck.

[@kylerjohnson1934]

Oh my gosh. The content is just too good on this channel!!! The perfect amount of proper useful nerdy tech, and absolutely hilarious execution. You guys are doing an amazing job. Thank you so much for the content!

[@77chrisray]

Huge fan Wendell. Thanks for all the help for us non computer science guys. You are awesome!!

[@post-leftluddite]

I would LOVE to see many more videos like this, specifically dealing with Networked Storage, raid arrays and enterprise hardware

[@ezforsaken]

"really popular, but canadian" Sorry linus

[@charleslowery9808]

OMFG the skit with Kreestuh at the end, amazing, and your storage array is surely the envy of much bigger youtubers, even LMG would be jelly.

[@ringlord13]

I'm not sure that I've ever commented on any KZbin video, but this was a seriously excellent blend of the high level project with hilarious cheesey nerd humor. Well done, +1 for fantastic entertainment!

[@goldbrick2751]

You guys are freaking awesome!!! Hell yes, Gotta LOVE Krista. All the best. Can wait to see more of this.

[@N1CH0LAS12]

Welcome to the club of more storage than brains club. I'm sitting here with 130TB redundant usable space :-)

[@txgamer1347]

I remember working on those PER710 Google servers at a Dell facility in Austin. Google required FLAWLESS paint and chassis, and would reject entire servers if there was one chip in the paint. They were also some of the most expensive configurations we worked out outside of the PEC Cloud series servers.

[@solitude335]

love the hacker typer in the background lol

[@djsensacion7]

He's watching the Internet...

[@1n50mn1ac]

Can't say that I understood much of what you said but you do sound like a really smart guy!

[@ldsudduthhanover]

I am literally rolling on the floor at that outro..blinking SOS...Ryan is a Clone..the KZbinr pictures, including Barnacules..... And I learned something.. Yippee!!!

[@aarongarza4769]

I always love watching Level 1. You all are great. Keep it up.

[@OhItsMulligan]

17:46 onward is LITERALLY GOLD. The sticky notes, the SOS from Wendell. 11/10

[@helchez]

Please don't stop. Love the humor.

[@hivijay999]

This was fantastic. I would love to see more videos like these showcasing server stuff. Maybe show us how to build a build server or something DevOps related?

[@frosty9392]

aww, look how young and yellow they are!

[@christoffer4017]

I'd love to see a follow up video on this, going into detail as to how it is done in practicality. It's all very fine with talking about it, but showing how it's done also helps in educating!

[@mpar]

11/10 educational entertainment. Level1 pgp cross over contexts. Cross channel tie ins. Great ground floor for building and tangenting content on. Keep up the great work.

[@tim9735]

did wendell blink sos at the end?

[@TommyThousandFaces]

Yeah, he totally did!

[@Aint1S]

Really excellent video, the best thing that TEK Syndicate ever did was to have the 🍺🍺liver🍺🍺 forcefully separate from the 🍸brain💻! This channel is like that new email account you created years ago without all of the unnecessary SPAM in-between the messages.

[@larrykozlowski6305]

I work for a LAN center during the summer and setup a Docker nginx cache using ZFS for all of the different CDN's for games about 5 months back. Think it was around 80TB for Steam/Origin/UPlay/BNet and another 10TB to cache media files from Netflix/KZbin/Facebook with a FiFo setup for when data starts getting full. It was actually a beautiful system when it was up and running and saved on a lot of bandwidth and also allowed us to let people download their games instead of being limited to the preloaded ones because we had that extra bandwidth headroom. I also setup a similar setup at home for caching my media and games just on a much smaller scale (5TB) and its actually pretty nice when you consider most of the people in my house will rewatch the same TV shows or Movies on Netflix, will download the same windows updates, and we all play the same games. Now instead of the 3 of us downloading a game 3 times, its just the one time. It is so nice lol.

[@nadpro16]

This is excellent, this is almost exactly what I have done with my personal file server at home. It is currently up to 60Tb with much more room to grow. It is nice to know that I think the same way in my designs. Now if only I could find a way to make money off it to. Because this is not cheap as a hobby.

[@NathanDavis2]

4:33 holy $$$$ that transfer speed. I can only dream

[@johnclark8542]

I think I just scorched my shorts. I

[@jorambahler8799]

That was really interesting stuff Wendell... glad you are up again with a fresh new team...

[@ryPish]

This is all I need in a video, the tech knowledge, the humour, the cringe, all the things in just the right balance!

[@cmusic52]

I swear with the antics I'm just dying for a LOVE button here youtube!!! Ssssoooo Hilarious and Awesome!!!

[@ryanbester7990]

The end scene was like a game... so cool. Also maybe put it near the beginning as a prelude for what is still to come haha

[@jokinboken]

We always knew who was in charge! :-)

[@TrueRegulators]

I have only watched the intro until this moment. Thumbs up!

[@eirajoao]

It's for things like these that I subscribe to you guys. You might have just won yourself a new Patreon backer as I don't want you to stop doing this. Keep up doing videos that are way above my head and, therefore, are amazing.

[@reallunacy]

This was fantastic. Question though, did Krista kill the other guy?

[@MichaelLindman]

What? Gwendell killed Twendell?

[@iboysven]

They will just clone a new Twendell

[@Level1Techs]

Nothing a tetanus shot won't cure. Plus wendell got to ride the rack like a skateboard.

[@MrSephkeene]

That was the best part of the video!

[@sykoteddy]

I can see the quality of the videos and studio has evolved a lot since 2017 :) But as neardy as always! I found you through LTT :)

[@simopr09]

how can someone talk that much without running out of things to say! pure dedication to technology

[@thatsux0rz]

I love you guys. Love all the videos so far!

[@Azivegu]

alright, if Ryan isn't in the next video then I'll be worried

[@bingliu2932]

It's ok, they can clone a new one.

[@NoNamenoonehere]

Krista Evil Genius page says we are currently on Ryan 4.6. due to fact Ryan keeps finding the secret lair.Wendell tho is original as he has been supplying Krista with Unlimited storage.

[@cataria3903]

omg the moment u realize the background on the levelonetechs site is interactive :o omg how do i get this gravity star super cool thing as fullscreen maxed out or as an application? what a beautiful distraction!

[@cataria3903]

only pity is that scrolling down makes the new star that gets created go off the screen :( not be created at mouseclick...

[@Level1Techs]

bug fixed :)

[@onkz]

Wonderful, you're very fast at fixing bugs!

[@cataria3903]

hm bug is still there. on firefox 64 bit, 50.1.0, multiprocess windows forced on, if that matters for it.

[@Jobucko]

nodegarden.js.org/

[@praecorloth]

Loving this video, can't wait to see a followup to it. Though I'm surprised to hear that ZFS is supposedly "bad" at handling IO for virtual machines. I've been running VMs on a pool of mirrors for years now. The only performance issue I've run into was when I switched from iSCSI to NFS. NFS, like CIFS/SMB, is terrible at tiny IO, and modern operating systems are loaded with tiny IO. Switching back to iSCSI was a nobrainer, and the VM performance is very smooth again. One of the things I did do, however, was disable atime. For VMs, it just doesn't make a whole lot of sense, so those operations end up being wasteful.

[@Nimitz_oceo]

This is some next level stuff! Level1 makes Linus tech tips look like a truck stop

[@antfry256]

Love the ending bit, Better start adding shelves. haha

[@pcsecuritychannel]

Outro: Twendell was cloned to perform such dangerous tasks! :D

[@oscargarciahinde4247]

Guys, you're nailing it! keep up the great work!

[@ipushpeople7221]

Wow, an informative, technical, intelligent and funny video. I'm always cool with the first 3. But adding the funny parts will be great for the channel growth, congratulations and !praise

[@tamask]

First I thought you were the same person, and I wondered how you could make the conversation so well in sync. Then I realized there really are two of you :D

[@AdamJee923]

Wow the level of acting 'scene' here in this video is breath taking lol awesome guys very entertaining

[@mathiswellmann6858]

This Episode was AMAZING!!! I love the backstory and the stuff around the main topic. Very creative

[@ThePeperich]

Hommage to "A beautiful mind" nice sidekick!

[@blackfireburn]

fav episode, i've been waiting for that for a while. good job keep it up :D

[@stonent]

Props for the Digital Unix box in the background.

[@kienanvella]

It's kinda hilarious (but extremely gratifying to know), this was pretty much the exact same solution (DS4243 disk shelves, LSI SAS 9201-16e, ZFS on Debian) I came up with for our data storage needs in our DCs at work, but we're using 12 bay R510s as our storage heads, and keep the boot disks in the internal bays - gives us some better storage density compared to the R710s. FWIW, these disk shelves are manufactured by Xyratex, and can take the same controllers as some of the Dell Compellent SAS6 controllers (model HB-SBB2-E601-COMP), which support SAS link aggregation.

[@AngateWier]

Thank you for this. I would love to see more in depth videos about the VMs and use case for this machine!

[@aarcaneorg]

I'm already subscribed, but I ain't gonna like or favourite until Krista does something about it!

[@RandoYoutubeViewer]

was funny but that whould be great to a see an in depth review of the choice you did make for the zfs pool and setup how much l2arc how is it setup you said you have a log device i suppose it the intel that do both l2arc and zil but hey that would be awesome to see

[@Level1Techs]

some info in the article, but not finalized on the setup. no l2arc devices -- just added more ram. we'd have to add at least 4 SSDs for l2arc to have it make sense with the workload. the workload just doesn't have a lot of random I/O except for the VMs which are small and use little storage. Right now It is two raidz2 VDEVs per shelf, 4 shelves, with a second zfs pool 4x4tb raidz1. Plus the NVMe for slog. We're experimenting with SSDs in the right hand side of the google enclosure. The 4x4tb as a zfs pool with two mirrors is plenty fast for VMs though. So this is not finalized yet but I'd expect to see a video about it on the linux channel before too long.

[@juliansbrickcity5083]

Level1Techs you said in the article you could loose a whole shelve but I do not see how your data could survive that. If you loose a vdev in a pool thats it, the whole pool is gone. Thats why I dont like adding vdevs to a pool thats on an second controller if it hasnt redundant pathways like your shelves.

[@SkylerHuse]

Is Wendell blinking in morse code at the end there? Loved that whole skit.

[@peterl4190]

You guys are awesome :p seriously the best tech channel in youtube. Keep it up!

[@neoskribe8986]

best ending ever.

[@XBoxSux]

Beginning of the video: "What?! No cutaway? So disappointing." End of the video: "That was everything I wanted and more!"

[@guillermo.gonzalez]

It'd be good to see some more detailed videos on the hardware setup and its parts and also on the ZFS tuning and why. For example, interested in see how you set up the multipath and the HBAs. Also, ZFS tuning that you mention on the article. Fantastic video BTW!

[@pauliewalnuts1024]

oh c'mon i was really waiting for a jumpscare there

[@CameronHeard]

I'm now referring to Ryan as Clone from here on out.

[@SimpleGunner]

somehow the first thing i noticed about wendells 'cave' was that hes has what im guesting is wallpaper engine running on the 2 top monitors, running some pretty interesting stuff. that ending skit was a blast.

[@mpar]

Taking the heated air from a pc, server, type situation and routing it through a custom mod old case or enclosure can be made into a jerky maker or fruit vegetable dehydrator. If that is something Level1 is interested in. I used to make jerky with sliced sirloin tip, high mountain jerky cure, wine marinade, spice drawer random additions, on a gas grill modified to minimize a consistent heat. Still have a piece that is 10+ years old and fine. Make captured heated air go through the enclosure and make level1 jerky.

[@MaelosV]

Thank you guys for this.

[@LessQQ432]

Can't wait till the linux channel picks up!

[@landwolf00]

Best channel on youtube!

[@gorudonu]

Really nice video. I liked examples you gave about usage. Please keep going!

[@monolel]

A great way of ease a very technical video... it reminds me AVGN type of producction.

[@Prodiod]

I feel like this is a "we're better" retaliation video to linus' server video about his 100TB server for Sevadus. However, this video is in good taste.

[@davidg4512]

heck yeah, been waiting for this video

[@ypey1]

the suspense, its like a James Wan movie.

[@93davve93]

Show us around in that FreeNAS build!

[@scooter4196]

Love your videos. Very technical and very precise. Thank you!

[@kyhldk]

a video on the steam docker thing would be awesome :D