ZFS Essentials: Auto-Converting Folders to Datasets on Unraid

[@Bluebirts]

Awesome work thanks as always. I did the script on the appdata folder everything worked, i did the same on the domains folder (three vms, Home assistant, retronas, windows 98) but just the retronas converted into datasets, the others are not shown in zfs master. the script runs trought without errors. zfs list shows that all vms are also datasets so i believe its a zfs master bug

[@SpaceinvaderOne]

Thankyou im glad you like it. the issue with folders for the VMs that have been converted (i would guess the datasets have a space in the name like "Windows 10") , it seems that datasets with spaces in their names are not visible or creatable in certain programs like ZFS Master. To fix this, you'll need to manually rename the dataset so that it doesn't contain spaces. You can do this using the following command: zfs rename "pool1/domains/windows 10" "pool1/domains/windows_10" After renaming the dataset, you will also need to update your VM template so that the VDisks point to the new location without spaces. I hope this helps.

[@resolutepixel]

​@@SpaceinvaderOne Hi Ed fantastic video i moved my domains to my secondary pool that is zfs and converted them to datasets. any issues with mover moving them to the array and back to my SSD pool once i convert that to ZFS?

[@resolutepixel]

I just answered my own question..mover does not move files (only folders) from datasets on zfs to an XFS array

[@glasshalfempty1984]

0:34 little details like this (starting out with the Master Z pool or whatever) can throw people like me off. I know that it is REALLY hard to keep track of everything when making an instructional guide. And sometimes you either don't think of something (most likely) or you think something won't matter. For people that have a hard time learning or people that overthink (also me), those little details can mean the difference between a simple watch and frustratingly re-watching over and over, replaying parts, and just generally having a bad time. I just want to share that. I'm not angry, I'm extremely grateful for your videos. They have helped me a lot. A LOT. You deserve more subscribers. It's just some feedback that you might not have considered. Or maybe you have, there's no way that I could know. Keep it up though, I literally only trust you for unraid video tutorials and I know you've helped myself and TONS of people. I explicitly look for your stuff when trying to figure something new out.

[@_TbT_]

Snapshotting (terribly important for ransomware protection) was one of the things we missed out with Unraid before (instead of a Synology NAS or True/FreeNAS). Not anymore! Perfect! Thanks for making that accessible for normal users. Unraid could and should however make some of the things you showed more accessible and should include them in the OS. Your videos truly rock and are a cornerstone for all Unraid users! 👍

[@SpaceinvaderOne]

Thanks for your kind comment and thanks for watching :)

[@kurosetsuna29]

Great video, thanks! A thought I had was instead of manually listing containers that you don't want to stop, you could instead list containers that have that normal folder mounted as a volume mount, then only stops those containers while it converts the folder into a dataset. Doing a quick dry test using ChatGPT I was easily able to get a list of docker containers that have a specific volume mount using `docker inspect` and `grep`

[@SpaceinvaderOne]

That is a really good idea. i had not thought of that

[@tommyjackson3811]

I'm in absolute awe of your generosity @Spaceinvader One!!! I have a fair idea just how much time that you give to help others in your detailed videos...... We are all blessed...... Thankyou for all that you do!!!

[@terrellclark]

Man without your videos on Unraid, I'd be lost. Loving the ZFS content, putting it into action at work and on my home servers. Cant wait for your video talking about ZFS replication.

[@geekdomo]

This is awesome SIO. I am eagerly awaiting your main array ZFS change over video. Nice scripts btw!

[@NicodemPL]

Thats a nice idea to move from Appdata Backup what takes quite long time (1hr for me) vs snapshots. This should be way more efficient. awaiting for next scripts :). Thanks!

[@andrebezuidenhoud3968]

Awesome tutorial Ed. Your content is extremely useful and that is why I am running unraid. I like the new intro.

[@shanezirkle]

Thank you for these videos! Super helpful! I wouldn’t have known about ZFS

[@SpaceinvaderOne]

Glad you like them!

[@Niklas983]

Thanks! Good to know is that if you have drives in the array formatted as ZFS the ZFS Master plugin will spin them up (incl parity) when the gui for ZFS Master is accessed (via the Unraid gui).

[@SpaceinvaderOne]

Yes that is a shame that happens. Hopefully the plugin author can prevent that in a future update?

[@gamer1xbox360]

Cool intro man, always looking forward to your videos!

[@SpaceinvaderOne]

Hey thanks. Glad you liked the intro. I like to try and change it up a bit sometimes:)

[@QuentinKingH2O_King89]

Can't wait for the next video for auto snap shot :)

[@reborn-network]

Awesome! was wondering how to do this. cheers for the effort in the scripts!

[@succuvamp_anna]

Is there a way to convert the main array disk? I used unbalance to move data off array disk, reformatted them to zfs then moved the data back but they're all folders, not datasets.

[@drmetroyt]

The background music - lets dance 🎉🎉🕺💃🏼

[@AlyredV2]

Excellent video! Thank you! going to be upgrading soon to try out some of this good stuff... especially with new NVME drives on the way... Couple of questions from a ZFS Noob: 1. Is there any penalty for the system to keep multiple snapshots? Coming from a VMWare/Hyper-V environment, keeping snapshots of disks can really suck up the disk space and reduce performance over time. 2. Can you go into a little bit on the difference between encrypting the entire disk/pool vs individual datasets? Anything to keep in mind in deciding which path to follow, for security or performance? Would an unencrypted disk with an encrypted dataset be theoretically easier to compromise?

[@SpaceinvaderOne]

1. Snapshots dont take alot of space but having alot obviously can creep up. Having a good retention policy for the snapshots is always good. The next video will show all of these things. 2. For security both LUKS and ZFS are really good. They differ in the attack surface they present. With LUKS, because it operates at a lower level (the whole disk), an attacker gaining access might be able to access all decrypted data. With ZFS encrypted datasets, an attacker would need the keys for each individual dataset,. So if the keys were different could be harder to access. On the other hand, if an attacker can compromise the ZFS systemexploiting a software vulnerability maybe, they may be able to bypass the dataset encryption. Also, ZFS encrypted datasets support encrypted send/receive for replication and can maintain encryption when data is backed up or replicated, something that wouldn't be possible with LUKS really. But really both are very secure

[@JasonEala]

As always … great tutorials.. thanks 👍

[@Holtonfamilyadventures]

Heads up the current Version 2 of the script ( as of 14th Sept 2023 ) has a bug for a few of us where it is spitting out an insufficient space error, Version 1 is still working fine though.

[@SteveCothern]

The error seems to be a trailing slash on a grep command in the script -- someone already filed an issue on this in Ed's repo, and I commented there about it.

[@MultiBS]

Thanks for the Video, really easy to follow as always

[@SpaceinvaderOne]

Your welcome. thanks for watching.

[@MinhTran-tj4wq]

The process was very clear. Though I still have questions such as if the files in the shared that was moved to the array was in 1 format and now that the new cache is for example ads with compression. When the data got moved back to the cache, does it now convert it to zfs and compress it?

[@RS-sd4cr]

incredible work ed

[@SpaceinvaderOne]

Thankyou so much. Glad that you liked it :)

[@666gene]

man these videos are good!

[@FlyingSucuk]

Thank you so much for your Contant.. as newbee to Unraid,.. it helps me enormous to get into it.. i would still dont know what to do with out your channel :D

[@SpaceinvaderOne]

I am so glad its useful.thanks for watching :)

[@SPXLabs]

I was there when this was being made!

[@SpaceinvaderOne]

No the pub was more fun. You would have been bored if i just edited the video and coded during your vist !!

[@kumper33]

Amazing! Keep it coming.

[@stonehenge313]

Great video, really helpful to me. I can see a lot of very cool things you could implement with ZFS

[@SpaceinvaderOne]

Yes ZFS has some great features :)

[@dgongut]

Awesome tutorial, it works fine at first try!

[@Apollopayne25]

Thank you so much for all your tutorials

[@SpaceinvaderOne]

your welcome

[@tazire]

@SpaceinvaderOne I just ran this today on docker appdata all went well from a conversion point of view. However, once the conversion to datasets was complete the script failed to restart the docker containers. Not a big deal but just means it requires monitoring. I don't know if mine was an isolated incident or if the script needs updating. Just said I'd leave the comment to let you know.

[@Gragorg]

This is awesome. Thanks alot for this video!

[@ChrisHolzer]

thx for the video, but i guess I will stay away from ZFS in unraid until they figured out how all that functionality can be nicely integrated into the GUI.

[@Terror-Gene]

My thoughts exactly. Seems like amazing features, but too much fiddling around for practical use. i feel like it would be too easy to forget whats been done & do something incorrect & mess up my data. I'll wait till some of these features are better integrated. Though the seperate datasets for appdata would be amazing! Much better than my 33GB CA_backup... what a nightmare to extract data from!

[@j-sebr]

Love the videos and can see that ZFS has so much potential. Hate to grumble, but any chance you could turn down the background music - I find it difficult to focus on what you're saying / doing. Keep up the good work, I owe you a beer.

[@musmuk5350]

Very interesting video Ed, thank you

[@SpaceinvaderOne]

Your welcome. Thanks for watching

[@guillermo1479]

What if I don't have any data sets but want to convert all folders in the zfs pool to a data set? EDIT: You just have to change source_path="${source_pool}/${source_dataset}" to source_path="${source_pool}" in the script.

[@JasonEala]

awsome work... but what if you have to replace the SSD? can you clone the drive with all of these configurations intact?

[@prongATO]

quick question, how do you set your view once you click on the contents of the pool/share? instead of the home icon and cache I have Index of /mnt/cache and don't get near as many details. I only have Size, Last modified and location.

[@FrankZlöbl]

Hi, great Video - great help as always! You rock unRAID ;-) Just one little question - to get the great Snapshot functionality, what do you think on moving the main DATA (spinning drives) from xfs to zfs? Automated Snapshots would be a great feature for Backups and Data security.

[@AlyredV2]

I believe he went over this in one of his previous write-ups (specifically for snapshotting) for patrons, but hadn't gone further into detail yet. I wonder if he'll be covering more of that in an upcoming video on backups and snapshot sync.

[@SpaceinvaderOne]

Yes stay tuned for more videos on all of this very soon :)

[@FrankZlöbl]

@@SpaceinvaderOne can't wait ;-)

[@39zack]

Just what I needed :D

[@rhino9094]

This was a great video, everything went very well, except for one thing. My binhex-delugevpn docker no longer works. The docker settings are good nothing changed there. The webui opens, but the preferences are blank and the plugins are gone. I even removed it and reinstalled it using the template. Same issue. Can't figure out what happened. Everything else works fine. Sonaar, Jacket, Handbrake, Calibre. No issues at all.

[@johannes5523]

Sensational script. It helped me alot! My zfs cache now works 100%. Thanks

[@destronger5313]

can this be utilized for backing up a hdd that’s in a zfs array? i have an additional hdd that i set up outside of my array that is like to make a snapshot of my entire array drive if possible.

[@Tintifaxize]

from my point of view this script is also very useful to convert any subfolder structure on another zpool - or am I wrong? Great Video, Ed - as always!

[@squeak751]

@spaceinvader one, I haave everything on a 1 tb Cache pool and now have everything set as Datasets. I recently bought some 2TBs to replace the 1tb. I copied everything over to the 2 TB and placed them in the same cache name as the 1s. turned everything on and nothing worked. No dockers and no Apps. I flipped the 1tbs back in and everythig went back up and working. So my question is how can i now upgrade my Cache Pool to 2tb and get everything to work?

[@905jay]

@spaceinvaderone thanks for the AMAZING content. Question, is it stafe to convert the array folders to datasets? I converted all of my array disks to ZFS as you instructed in another video and everything is running smooth. Also, I converted the cache as instructed. Is it safe to do the array or is that another topic for another video?

[@IEnjoyCreatingVideos]

Great video as always Ed! very helpful :) Thanks for sharing it with us!💖😎👍JP

[@SpaceinvaderOne]

Thank you

[@IEnjoyCreatingVideos]

@@SpaceinvaderOne Your very welcome Ed! Have a fantastic week!😎

[@icenode]

Am I right in saying that after converting appdata folders to datasets, if we delete a Docker container in the UI, now we'll also have to remember to destroy the appdata folder manually/separately?

[@BLKMGK4]

I have a large amount of data for VM and for Appdata, I keep it on a drive other than my Cache drive mounted using Unassigned Devices. Is it possible to convert drives mounted this way to explore the benefits of ZFS and use Snapshots? Currently I don't get things like my Plex metadata backed up as it's too difficult with so many small files but a snapshot would be perfect! How much disk space do snapshots done like this take up? Stored on the same device correct? If you lose the device this isn't a true backup right - can the ZFS snapshot be done to a second device mounted with Unassigned Devices? Maybe even a remote share? Hmm!

[@ghstfce]

Excellent video per usual. Though i can't understand why these functions aren't present natively.

[@NicodemPL]

Script v2 always gives me this errors: Skipping folder /mnt/cache/appdata/XXXXX due to insufficient space . Any idea why?

[@walterschoenmann5162]

Hi SI1, I see you have your "appdata" folder separate from your system folder on your cache drive. On my cache I have my appdate default location pointing to my "system" folder. Is that a good thing? should they be separated. Your advice would be helpful.. Love the videos..

[@AlyredV2]

When moving an appdata folder back from the array to the new cache pool, can we manually create the datasets making their names match? This would save time in having the mover script do the work after the fact. Will this confuse the FUSE system in any way?

[@derfacecrafter1869]

The script works, and every docker is its own dataset. But on the array, have no dataset. I know why I can't use the 'folder to dataset' script, but not what I should do now.

[@TunTin-yv3qq]

How did you enable all the buttons in the > user > appdata (jobs, search, delete, copy, rename.....)? I only see Index of /mnt/user/appdata and no buttons. I am a newbie here - thanks much.

[@mattym00]

Dynamix File Manager kzbin.info/www/bejne/npuYlaateLmnbNU

[@na1du]

After upgrading to beta 7, do you still need to do this?

[@w.schobel1514]

Thanks a lot. That is amazing!

[@SpaceinvaderOne]

Glad you liked it :)

[@ClanLawrence]

Awesome video. Looking forward to the next one. Does this affect the function of the "CA Backup/Restore Appdata" or "CA Cleanup Appdata" Plugins? Also, how do we get VMs to autoshutdown when doing this with the 'Domains' dataset?

[@SpaceinvaderOne]

No the backup plugin will work just the same. To shut down vms set the vm variable to yes in the script. One thing to note is zfs master doesnt see datasets with spaces in the name. The script will see them and convert them correctly but after if a space zfs master will not show them. Please see the pinned comment at the top and my reply

[@sballerstedt]

Great video, thanks to your instructions, it's always easy to implement such things. Many thanks! I have a question though, in the dashboard, under System, ZFS at 100%. However, my two cache drives are not full. Does this have any significance? Cheers.

[@Sharko187]

Same and i have no idea why :D

[@resolutepixel]

Super helpful Anyone follow the previous video and have issues with the Arrs and Plex? The *Arrs database was gone once I was on ZFS and Plex container started but logs complained of sqlite 3 access error. Docker image was btrsf vdisk but tried XFS vdisk and by directory none worked had to format and set my pool back to BTRFS and everything worked I set it back to

[@phileeny]

So I have no clue, I used the last video to setup a pool, and this video to setup datasets, eg, foldername/folder1/folder2 using zfs master, then went to the folders in windows, great I see the folders, to test the snapshot I dragged a file in to folder2, what? I don't have permission, tried folder1, no permissions, so I tried right click to add a new folder that worked but can't rename it, I've looked through the video again, did I miss a step, there is no talk of permissions, help!

[@tknx5909]

Why not combine the two scripts - have it identify any folders in the watched folders that aren't data sets and convert them instead of having to explicitly list the ones that are already done?

[@tknx5909]

Also - how do you have it run on your vm folder and appdata folder both?

[@ICHIMOKU77]

Is there anyway to revert back ? This script fucked up my appdata permission

[@BillxSand]

I love this but i have a bunch of pools with a bunch of datasets that I want to protect. How do I do this automaticaly en masse instead of just one share at a time?

[@RafaelSilva-yv3oh]

there's a section of the script that says: # Additional User-Defined Datasets # Add more paths as needed in the format pool/dataset in quotes, for example: "tank/mydata" you can probably add them all there and then run it

[@tknx5909]

Script gets stuck at the first covnersion for me. I left it overnight and it didn't go anywhere.

[@Sharko187]

Where the snapshot will be saved, is it in the same cache pool?

[@Sharko187]

If yes what will happen to the snapshot if, i relpace the ssd cache drive.

[@ЯрославСидорович-т1с]

maybe someone have issue when rolback dataset all shares disappears and need to rebot server

[@veizour]

Mine keeps giving "due to insufficient space" even though my appdata is 93GB and my free space on that drive is over 600GB

[@comp20B]

I get this same error for many folders. Any update?

[@Holtonfamilyadventures]

yep same here.. @SpaceinvaderOne is this a bug in the script?

[@SteveCothern]

The error seems to be a trailing slash on a grep command in the script -- someone already filed an issue on this in Ed's repo, and I commented there about it.

[@comp20B]

@@SteveCothern Thanks! I removed the slash, tested a dryrun and then ran the conversion. It worked perfectly!

[@acrusso1]

this worked but the script froze twice - killed it re-ran it seems to have gotten everything ( i hope)

[@Mavrag]

Hey mate, thanks for the video. If i have a truenas scale server, is it possible to boot from a unraid usb and open zfs pools that were created in Truenas?

[@SpaceinvaderOne]

Sure! It is possible to import zpools that have been created on TrueNAS. You can choose the number of drives for the pool and let the filesystem be set to auto. Once the server starts, it will attempt to import the pool. However this may not work for all zfs config but most regular zpools should work

[@IanW6]

A god among men.

[@SpaceinvaderOne]

Thanks for the divine compliment! :)

[@jonirish323]

Great Video, but what about VM's? Your scripts "sees" them, but I don't see datasets being created for them.

[@SpaceinvaderOne]

I have pinned a comment to the top. Please read my reply there i think it is why you may have this problem

[@jonirish323]

@@SpaceinvaderOne Ed, I went backed and looked, and none of my VM's have spaced in them, but they still don't show up as ZFS datasets?

[@romankiss3793]

amazing, thank youi

[@SpaceinvaderOne]

Your welcome :)

[@andis8392]

hey, really nice video. I started to convert all my cache pools to zfs and wanted to convert the top level folders to datasets. Worked on a zfs pool with a single device fine. On my zfs raidz1 pool (3x 1TB) for my Nextcloud the script ran without any issue except it shows me now that the free space is gone. Before I ran the script there was 50% free, now only a few GB are left free. I don't see any "xxx_temp" folders in the filesystem anymore, so the cleanup should worked fine. If I calculate the size it shows only "total occupied space: 938 GB" The zfs pool shows size of 1,9TB and 1,89TB used. The zfs master shows size 2,72TB and 2.58TB used. Any idea why this happened and what I could do or check? thanks

[@andis8392]

I used the zfs raidz pool for my nextcloud for a few weeks now and it worked fine, before I ran the script to convert the folder to datasets. Turned out I was not able to create new files or folders within nextcloud as a signed in nextcloud user, which worked fine before I ran the script. But I was still able to create folders with Unraid file manager within the nextcloud subfolders. So I decided to destroy the datasets one by one to see what happens to the used space. Surprisingly the used space went back to 938GB like before and not to 0, what I was expecting. All the data was still there and was accessible within nextcloud and I was able to create new files and folder again as signed in nextcloud user as if nothing happened. Still not sure what exactly happened, but it works now as before. I will start a new try with the script on my nextcloud folders but will run it only for one user at a time and after I did the first one I will doublecheck if I can create files and folders as signed in nextcloud user.

[@RussellHarrisAus]

Not all my cache folders are showing in zfs master or zfs list, including appdata, even though they are on my zfs cache and all working fine. Also a dry run says no space available even though the cache has twice the free space available as compared to used.

[@transatlant1c]

Try asking in the unraid forum, more likely get a useful response

[@TheSpantsutube]

Have exactly the same problem. I created a github issue

[@SpaceinvaderOne]

Run the list folders dataset script to check the parent folder is in fact a dataset. If they are not showing then the share may not be a dataset

[@AquaRelliux]

I also have this problem after following your guides. How do I make the parent folders into datasets if they are not?@@SpaceinvaderOne

[@oakfig]

Do they have to be datasets instead of folders in order for snapshots to work?

[@calbraun1668]

w00t!

[@jonjohnson2844]

Aren’t we allowed to called it “zed”fs as Brits 😩

[@Theexplorographer]

Going to be honest here...lately your videos have been less than helpful. The videos have been great, but as a medium level user I have not found any of them useful. Would love to see videos on how to set up things like ViewTube or some of the Discord bots out there. Or updated guides on Swag, Nextcloud, etc. Sorry if this upsets you, just being honest here.

[@SpaceinvaderOne]

Hey no worries man. Appreciate the feedback. Yeah its difficult for everyone find each video useful. But yes i may well make a video on some of the topics you mentioned :)

[@IanW6]

Going to be honest here...your comment is less than helpful.

[@Theexplorographer]

@@SpaceinvaderOne Thanks man. I will still continue to watch and support your work and appreciate all you have done for the community.

[@Theexplorographer]

@@IanW6 I wasn't talking to you so thanks for your opinion.

[@Th3ch0s3n0n3s]

@@Theexplorographer I think your opinion is fair, but you also should understand that some of your desired videos (like updated versions of old videos) have GOT to fall second to new content such as this. ZFS is brand-new to unraid, and it comes PACKED with features that are useful to ALL users - not just the hardcore users. This video is a prime example, as I've had several times where I've fucked up my plex appdata, and wish I could just click a simple button and go back to where it worked and NOT do the thing that fucked it up.

[@drrobotsir]

I have 4 folders under appdata, and the last one failed to clean up after running your script. It did create the dataset, and somehow the source total size shown in the log file is actually smaller than the actual size shown in file manager. Here is the error in the log: Validating copy... Validation failed. Source and destination file count or total size do not match. Source files: 587, Destination files: 587 Source total size: 27948074, Destination total size: 32198443 Any ideas? Thanks!

[@samubolvivat9504]

i lost all docker in dataset folder when using this script. zfs dataset things is so confusing