ZFS for Newbies

[@FunctionGermany]

when i was new to ZFS, i didn't understand the difference between pools and vdevs too. vdevs is where you implement redundancy, either through mirroring or parity (raidz). multiple vdevs is how you expand the usable storage of a pool (until raidz expansion ships). if any vdev fails, all of your data from the pool is inaccessible. besides expansion there is a second reason for why you might want to create a pool from multiple vdevs instead of a single large one: raidz resilvering (i.e. after replacing a faulty drive) is extremely straining on all drives of the faulted drive's vdev. if you put 20 drives into a single vdev for a pool, resilvering will add wear to 20 drives. if you instead do 2 vdevs with 10 drives each, a resilvering will only add wear to 10 drives etc. so vdev arrangements are also relevant for wear and risk management.

[@bingolio]

Thank you for the best practical explanation of ZFS, even 4 yrs later!

[@Dutchlincoln]

As a die-hard windows user, even i understood 80% off it... :) Just bought me a Qnap NAS (one with zfs, as my son advised) but there were some parts not clear that are explained now. Thank you. (pitty a raid set cannot be expanded, only second dev added... costing one extra disk...) Great explanation, thank you.

[@just1689]

I think they have expansion now?

[@hadeseye2297]

"die-hard windows user" Should be: die-hard spyware user

[@Austin-xx2nu]

Single drive expansion was just added very recently.

[@bertnijhof5413]

Nice video and to support your claim at 24:57. If have a ZFS system running at 1.25 GB of DDR (400 MHz). It runs FreeBSD 12.1 on a Pentium 4 HT (3.0 GHz) in a Compaq Evo Tower with Windows 2000/98 stickers. The motherboard is from a 2003 HP D530 SFF system. All leftovers, I only paid DOP 250 ($5) for that old tower and DOP 800 ($16) for a new 500 Watt iTech power supply. The power-supply has no video card connector, 2 SATA connectors and 2 Molex connectors. The PC has the following HDDs (1.2 TB): - 3.5" IDE 250 + 320 GB - 2.5" SATA 2 x 320 GB on a SATA-1 connector :) - The PC has two cables Power and Ethernet. FreeBSD is running xfce, xrdp and conky too. It is in use for

[@bertnijhof5413]

@@CJ-ff7rr Why not! It was cheap and it is in use now for 2.5 years.

[@georgH]

I also have such server on another room, just works 1h per week like yours. I am using: - znapzend to automate snapshots (creation and deletion of old ones) and baking up to the other server - Wake on Lan to turn on the backup server - mirror on the backup server, just for good measure (I want the backups to be reliable) - Have been running it for 3 years, 2 years unattended (was living abroad) and... worked without issues! - mbuffer is used with zfs send - In case of power failure at home, the backup server is set up at the BIOS to always start. Then, Linux sets up the WOL and shuts it down after 15 minutes of inactivity :)

[@quistian]

Thanks Dan for this informative video. I use it when I am training new IT people who are using FreeNAS.

[@JohnNada-b9s]

11:15 they've been working on that for as long as I remember and it's still not there. The inability to reshape pools is a serious hindrance and the reason I won't recommend ZFS to anybody for as long as it remains so. Having to move all the data out, destroy a pool, recreate it and move the data back is dangerous, expensive, time consuming and as fun as a root canal.

[@Our_Sole_Pusch]

Man, I'm glad this was for newbies. I'd be completely lost by now if it wasn't.

[@DanLangille]

Was it good for you?

[@MaulwurfMedia]

this is a great video, thanks for making it. I would love to see a "indepth" video for newbies like me who want to dive deep into it :)

[@samramdebest]

25:20 can someone explain what he's saying here? it doesn't make much sense to me. To me it sounds like he says, only put your os on zfs and your actual data on other non zfs filesystems. But this is a zfs talk so that doesn't make much sense. Then I looked up UFS and found unix file system, so he's saying to put like a virtual filesystem on top of zfs and boot from that? and what is the HBA he's refering to? also booting from motherboard?????

[@KaliszAd]

Host Bus Adapter is usually a card or a special part of a server mainboard that gives you more ports like SATA/ SAS or even NVMe these days. There are also port replicators and backplanes usually for SATA/ SAS drives. Because a single SAS connection has so much throughput capacity, you can split that up with a bit of help from the port replicator and e.g. attach 12 disks to a card, that could directly only support 2x4 ports. This setup is quite common on 2U server chassis. Similarly, there are up to 90 disk Just a Bunch Of Disks enclosures that you can also attach using just a few connections. UFS is a legacy filesystem that you would historically use instead of ZFS. Today, you can use ZFS also for the boot drives, no need to have any UFS.

[@FrankZambaras]

This guy is the Matt Dillahunty of file systems. 😀

[@DanLangille]

Because we both have shaved heads and a beard?

[@FrankZambaras]

@@DanLangille Sort of. But it's more than that. I see a striking similarity. Anyway, no disrespect intended to either of you fine gents. Much thanks for the excellent content. You helped me get my ZFS Pools spun up.

[@OfflineSetup]

@@FrankZambaras I was thinking the EXACT SAME THING , not because of the appearance, but because of the vocal tone, and (more importantly) the "forthright" dismissive attitude. I don't say that as a negative.

[@anirangoncalvesbr]

I was thinking the same. I'm just about to start a ZFS pool and I ALSO happen to watch a lot of Mr Dillahunty videos lately 😊

[@FrankZambaras]

@@anirangoncalvesbr Glad it's not just me!

[@GizmoFromPizmo]

Note to the watcher of this video: when you hear the word "fall-system", understand that the presenter means to say, "filesystem". Fall-system doesn't have anything to do with a system of falling.

[@rogernevez5187]

18:17 Imagine if it wasn't for newbies ....

[@kenwood7195]

Yes, it's looking like a specialist field.

[@RedFenceAnime]

Maybe I misunderstood but it looks like he's mounting a drive (or zpool) in a folder on another drive. Like how you can have a drive for "/" (root) and another drive for "/home/" which is mounted in root. A bit like how Windows users put their documents on D drive so if windows crashes and takes out the C drive it doesn't take their files with it.

[@DanLangille]

@@RedFenceAnime Yes, that's right.

[@MaxUgly]

This guy is the Ben Stein of file systems.

[@DanLangille]

My dry humour?

[@MaxUgly]

@@DanLangille yes it is my favorite type of humor! Dry eyes on the other hand are terrible. You both are awesome people.

[@Guilherme-qk9so]

this was awesome, thanks for sharing!

[@glasshalfempty1984]

It's funny that you call this ZFS for newbies...I still picked up a lot, but I'm going to need a lot more videos.

[@shibikumar5473]

Great video.... Perfectly designed for a newbie.

[@nicolasborn3788]

16:54 three or more drives for raidz

[@JimGriffOne]

Can Windows 10 run on ZFS instead of NTFS? If so, I'd like to use it to make a big backup drive with 4x 14TB hard drives.

[@alexanderkrow3566]

No, it can't. Microsoft refuses to support any other filesystems than NTFS and the FATs.

[@lithiumflower31337]

Your best bet would be to use another computer and access the disks over your LAN. Either using BSD and zfs yourself or something more turn-key like TrueNAS or some other off the shelf NAS solution.

[@petrusion2827]

I know this comment is 2 years old but.. for Windows you could look into REFS, it seems to me it is (or at least it is meant to be) Microsoft's alternative to ZFS.

[@judclark7376]

I love those purple chairs

[@goldibollocks]

One thing I still don’t get about snapshots: They are immutable but when you delete a file in the live dataset that was present at the time the snapshot was taken, the snapshot retroactively gets bigger. Isn’t that being mutable? And how does another host, where you transferred that snapshot, get the info that the snapshot is now bigger?

[@lithiumflower31337]

I think you have snapshots a bit backwards. They aren't 'archives' of information that gets bigger when something is deleted. When a snapshot is created meta data of what blocks are required to restore that state is tagged on each block, defending it from modification. If no modification occurs, no new space is required, that same block can be read from when using the live datatset. When the live datatset needs to modify something (whether that's adding to or deletion of a file) that area of space is left as-is and a copy is made to a new block and modified there. This is referred to as Copy-On-Write or COW. If that other host is keeping a full copy of the system then only the difference between it and the last snapshot would be sent. If it doesn't know anything about the system, then all the blocks needed to recreate that snapshots' state would be sent over to it when you do the transfer (which would take longer). Any further modifications on the 'live' system are, well, live. They haven't been backed up/snapshotted so nothing about what happens afterward is relevant. You should think of it more like layers in photoshop, when you snapshot you make that layer read only and start doing new work on a new layer until you want to save that state again. Anything that didn't change is transparent and you see the original below (or any modifications made on layers between that one and the original). Deleting a layer/snapshot would be more like flattening one layer onto another, everything that's the same won't change but the newest info/higher layer will overwrite the stuff below it. So sending that to a remote host would be like flattening that layer down to the bottom and sending the image as it looks like from that layer.

[@FunctionGermany]

snapshots and live dataset are layers. a file and it's changes are attributed to the lowest layer (if you imagine oldest snapshot at the top, then newer snapshots, and live at bottom). lets say you have 10 files, 1MB each in live, no snapshots. live uses 10MB. now you make a snapshot, it takes 0MB because live has all the data and the usage. now you delete 1MB from live. it now uses 9MB and the snapshot is the only thing holding the "deleted" 1MB, so that's it's usage, and the snapshot didn't get modified in terms of what data it references. this is the same reason a new snapshot starts off as 0MB large. it's size is equal to the amount of data that differentiates it from the lower levels that tend to still reference most of the data. when you now remove all files from live, the snapshot is 10MB big.

[@om_mani_padme_hum]

thank you

[@dhakshinamoorthy.b]

Hi Dan, Hope you're doing well..! I'm planning to use a commodity server with Ubuntu OS + Block Storage. Server will have 2x local disks for OS+Boot and also, it will have 4x NVMe drives. Storage will have 40x 18TB NLSAS disks which can provide 500 TiB of storage capacity post DRAID6. Planning to deploy "OpenZFS" and export 500TiB as NFS share for storing large size images. In server, 4x NVMe will be used for both L2ARC (2x NVMe) and remaining 2x NVMe can be used for ZFS's ZIL/SLOG Backend storage volume mapped to the same server and part of zpool We will apply large block "recordsize" 16MiB since we need to store image files larger in size ~100GiB, ~150GiB etc., I just need your inputs, can we create a "zpool" with backend storage volumes/LUN that is already RAID protected by backend storage array?

[@vladx71]

this great! Thanks

[@JasondeCordoba]

this reminds me why I miss techsnap

[@DanLangille]

I enjoyed TechSNAP, but I gave it up because of the workload it involved. I wanted more time for other ventures.

[@JivanPal]

I knew the voice was familiar!

[@kanikabhatia4171]

What was the answer to recover from read only snapshot?

[@Drewcey]

The stored snapshot can never be changed, but when you do a send/receive of the snapshot to the new location, it becomes read/writable on the receiving system. The original snapshot still exists in the backup location, so you're essentially just sending a read/write clone of the data.

[@galen__]

30:40 - I think of zpools as “The Pools of data”, or “Zee Pools of data”, whereas I think of the Z in ZFS as first, the Zetta unit of measure (Zetta = Zed), and second as “The File System with The Pools of data”, or ZeeFS. I guess ZeeFS refers to ZedFS, so both can exist and work 😁

[@ericjbowman1708]

"Math is hard, but they're working on changing that." I love quoting folks out of context lol

[@MrAtomUniverse]

Wait , you mean if i have ZFS enabled, and my NON ECC ram screws up, it wont screw my data? The worse that can happen is maybe the computer crashes and needs restarting? Is that what you're saying mate?

[@JivanPal]

Yes. The checksumming ensures that only intact data is used. If any file metadata/data you try to access is malformed, ZFS will log this and attempt to access an older instance of that data from a previous transaction. However, other disaster scenarios can occur that don't involve RAM failure, such as failure on the interface to the drive which results in blocks of data being randomly clobbered. In that case, it is completely possible that existing, past transactions are corrupted, in addition to the latest transactions.

[@georgH]

Actually, what he says is "ZFS without ECC is better than no ZFS". If your system has wrong data in memory, and wrong data is written to disk, that wrong data gets checksummed as usual, ZFS can not now whether your data coming from memory is good or not. Upon retrieval, ZFS will check the checksum and it will be fine, you will get exactly the same you put in earlier, which was already wrong.

[@grzegorzk1177]

Fine for me but... why do you present the diabolic picture. Do You think is not enough devils acts on its self on the earth?

[@ft6755]

BSD convention and he's using a Mac?

[@blckhwk8024]

MacOS came from BSD if 'm not mistaken, so it's not that bad :). Also FreeBSD is not so good for desktop.

[@TCHICKIBRAXD]

Wait, what?? GhostBSD is awesome for desktop

[@theantipope4354]

The recent versions of IOS are BSD.

[@DanLangille]

Use whatever tools you like. I use both open source and closed source tools. I have run FreeBSD on my laptop, I've used Windows. Right now, I'm using OSX.

[@ps5hasnogames55]

Yeah you notice that a lot with FreeBSD developers. They love FreeBSD so much, they use Macs instead. FreeBSD is developed in and for virtual machines only.

[@llortaton2834]

ya boring, and thats after watching 5 vidsabout zfs, the content that you teach is good

[@LincolnSharon-l7c]

Smitham Glen

[@MuminovicGoran]

zed :)

[@darren537]

Pretty sure I have a severe learning disability at this point

[@ricsip]

the presenter MAY be a guru of what he is doing in his daily job, but for sure gave a terrible difficult to follow presentation. References to things the audience may not able to follow is a terrible choice for presentation that was clearly for beginners. This presentation didnt have a clear starting point and a clear ending point. It was jumping from one random topic to a different non-related topic in the next minute. Arrogancy part when he was asked about questions he didnt want to answer is also a big minus on his personality level.

[@ErikS-]

He is probably a very knowledgable guy, but presenting and answering questions is a mess.

[@FreemanEleanore-b1v]

Thompson Sharon Miller Angela Johnson Jose

[@TylerDeBoy]

Coronavirus present in that class. Every minute someone coughs

[@HiltonT69]

Oct 2019 - SARS-CoV-2 wasn't around then...

[@DanLangille]

This was recorded well before COVID-19 became a thing.

[@WarnerVincent-g2c]

Robinson Robert Perez Steven Thomas Paul

[@MrArmas555]

++

[@ReneeBryant-o1e]

Hall Scott Johnson Nancy Clark George

[@FoxMeredith-b4y]

Garcia Angela Garcia Daniel Jackson Michael

[@DaisyHayes-n2k]

Moore Dorothy Hernandez Jason Martinez Lisa

[@SusiePerez-d7o]

Jones Matthew Wilson Timothy Miller Shirley

[@KarenLee-s6x]

Robinson Larry Thomas Jeffrey Garcia James

[@LeonardRiley-y6z]

Moore Kevin Jackson Jennifer White Deborah

[@AlyssaReynolds-u7i]

Johnson Brian Walker Angela Martinez Lisa

[@LindsayIna-r2t]

Allen Ronald Hall Donald Robinson Sandra

[@catherinecarroll6462]

Walker Edward Anderson Mary Brown Donna

[@NoyesBruce-k4n]

Harris Maria Moore John Young Jose

[@DoraSpring-m9o]

Thomas Steven Hernandez Margaret Lee Laura

[@KiplingGeorgia-e6p]

Gonzalez Melissa Walker Dorothy Thompson Matthew