Mastering ZFS: Easily Auto Snapshot & Replicate a ZFS Dataset - Part A

Effortlessly set up automatic snapshots and replicate a ZFS dataset on your Unraid server with the provided script. This not only facilitates quick autosnapshot configurations but also allows you to replicate (or backup) your data to another dataset, be it locally or over a network or the internet. If you don't have another ZFS location for backups, don't worry. The script can manage that as well. Simply toggle to rsync replication, and your dataset will be snapshot-taken and incrementally replicated using rsync to any desired location
Please, if you can and want to support the channel and donate you can do so by Paypal here goo.gl/dw6MLW or check my patreon page / spaceinvaderone
Bitcoin donations 3LMxDzcwPdjXQmmeBPzfvUgjYFDTqDAgQF
Snapshot and Replication script
github.com/Spa...
Beep speaker
Amazon USA - amzn.to/3swqv6Y
Amazon UK - amzn.to/45qFrlR
----------------------------------------------------------------------------------------------------------------
Need a VPN?
PIA is popular with Unraid users as its easy to setup with various vpn download containers - www.privateint...
Torguard is also an excellent VPN again with both openvpn and wireguard protocls supported.
Get 50% off for life using code spaceinvaderone torguard.net/a...
----------------------------------------------------------------------------------------------------------------
Need a cheap windows 10 license for around $10
consogame.com/...
----------------------------------------------------------------------------------------------------------------
Need to buy something from amazon? Then please use my link to help the channel :)
USA - amzn.to/3kCikfU
UK - amzn.to/2UsYb1f
USA link - USB HDD Docking station amzn.to/3v754WG
UK Link - USB HDD Docking station amzn.to/3hLenYp
HighPoint RocketStor 6414S amzn.to/3fiXv9s USA
Mini SAS 26-Pin SFF-8088 Male to Mini SAS 26-Pin SFF-8088
amzn.to/2V4x9kT USA
amzn.to/3xfkxEl UK
----------------------------------------------------------------------------------------------------------------
A big thankyou to Limetechnology for all the great work that they put into always improving Unraid OS.

[@ulfthomasjohansen5000]

I just love your videos sir! Easy to follow, very structured and to the point - each and every one. I have just followed your latest series on ZFS and have updated my cache pools to ZFS, installed and scheduled your scripts and even gotten replication to work against an RPI I have in a corner in the garage. Works perfectly! :) I do have one question: If one, for any particular reason, should loose the disk pool that holds the snapshots how does one recover from the off-synced snapshots? It might be clear as a day to others, but not to me. :) Thanks again.

[@905jay]

Thanks! Your videos are incredibly helpful and I appreciate all the time and effort you put into it

[@905jay]

I've converted all my array disks to ZFS. Does it make sense to now convert folders on the array to datasets? Does it make sense to convert especially important array folders, all array folders /shares, or none at all? And if it does make sense to do so, can I use your dataset creation script to do it?

[@chillstar]

@@905jayEd goes through the difference between folders and datasets and the benefits of datasets in his recent video ZFS Essentials: Auto-Converting Folders to Datasets on Unraid: kzbin.info/www/bejne/pn2pknekibujq68 If you're a beginner with ZFS (as I am) it's worth having a watch through the whole ZFS playlist: kzbin.info/aero/PL6MCtOroZNDCr7TgKEgYDD_5WPV75M2If

[@SpaceinvaderOne]

Thankyou so much for your tip :)

[@icenode]

Mirrored app cache drives, replicated to the array including snapshots thanks to this video, and backup up to the cloud. If I somehow STILL lose my appdata, something really has gone wrong, haha!

[@SpaceinvaderOne]

Definetly. If you you loose your appdata now you were just never meant to have it lol ;)

[@rjbrowning85]

With the replicated snapshots and the backup to the cloud, do I still need to mirror my cache drives? Or can I get the extra space back?

[@PeterCarlsson-k7n]

Hi @icenode , how did you solve "and backup up to the cloud"?

[@chillstar]

@@PeterCarlsson-k7n I went with Backblaze B2 for inexpensive online storage and the Duplicacy Docker to handle the synching, encrypting and restoring. Been working well for me for a year now.

[@Harjinder.Australia]

what if you want to snapshot multiple datasets in the same source pool? comma separate them?

[@MikeR-gc6ir]

I really with this was addressed. I found a comment on his github from a long time ago and also doesnt have a resolution. I don't want to risk my data trying it out even though its super simple to see if it works. Ive got 5 different scripts running at 5 different times do backup my various zfs cache drives/directories. I would love the option to just use a comma, and run them all together so I didnt have to balance potential time conflicts of 5 scripts..

[@Frazzer951]

For some reason it isn't deleting the snapshots on my backup drive. I have 13 snapshots on my cache drive with the appdata. While I have 24 on the backup drive.

[@Jleonard3314]

I'm having the same issue, I would be interested if you found a fix or not.

[@Frazzer951]

@@Jleonard3314 I created a PR on the github to fix it. You have to also add the flag --delete-target-snapshots to syncoid to delete the snapshots

[@olitrolli8695]

Same issue! Would be great if someone could share a fix for it.

[@Frazzer951]

@@olitrolli8695 The fix has already been added to the GitHub

[@tomlasparini9353]

I'm not sure if this is related, but I think once you run the script for the first time, changing the retention parameters in the script doesn't have any effect. This line I think says that if a sanoid config already exists then don't create a new one. Well one would exist the first time you run it and then never change again no matter what you change the retention variables to in the script. I had to manually edit the sanoid.conf file. # check if a configuration file has already been created from a previous run, if so exit the function if [ -f "${sanoid_config_complete_path}sanoid.conf" ]; then return fi

[@aarongibbs2479]

How did you get the "Main" tab separated into tabs for Array Devices, Pool Devices, etc? I like this so much better than having to scroll up & down.

[@lovelysandhu7790]

Go to Settings->Display Settings and change Page view to tabbed.

[@RafaelSilva-yv3oh]

@@lovelysandhu7790 Thxs!!

[@scottmielke4071]

Amazing. I wanted the benefits of a ZFS pool for running special vdevs and a slog with some speedy network access and then wanted a once per week sync to my array, this is great

[@prongATO]

Maybe it's pretty self-evident but could you possibly do a quick restore from the backups? single container, multiple containers and the whole appdata folder.

[@laov6843]

I agree. Maybe it is just as simple as copying everything from disk13 into your new - empty cache pool under the same names but it would be nice to mention how it would be done.

[@genghistron1858]

@@laov6843 Same here. I added a new 1tb NVME drive to my ZFS cache and it wants to format both drives despite being set to mirror. I can either run off the original NVME or format both and restore snapshots, but the options don't make it clear on what is doing what.

[@x1mega]

I'm in same boat here. I'd wish to see how you use the snapshots which are stored in the array. Do these need to be moved back to original disk/share or can they be used straight from backup share?

[@genghistron1858]

@@x1mega from what I can tell, you should be able to use them from the backup share. I've had some successfully restore and some not, so there is still something I'm missing.

[@rylangrimes88]

I am in the same boat. want to swap cache drives. but shouldnt you just be able to copy the data from the backup folder and start docker?

[@KNIGHTER1995]

How on earth did you get tabs on your Unraid main area 13:49 looks so much better than Unraid’s traditional “here’s your 30 disks”

[@Gex2501]

It is in the display settings.

[@Gex2501]

Specifically Settings->Display Settings->Page View

[@prongATO]

tabbed view

[@Nightowl80]

Thanks!

[@MarcHeyer]

Thanks for this great series. I finally came around setting everything up and it works like a dream. The V2 of the Auto Conversion script is also really great, especially the skip when the destination is already a dataset. I have some questions: - I´ve set up one script per dataset i want to snapshot and replicate. Multiple datasets pers script are not supported or did i miss something? - What is the best strategy for offsite backups? I have a second unraid server 500km away and would like to replicate to it too. Do i just set up every script again, but with the remote server? What source pool do i choose in this case (the one on my cache again or the one in the array)? If i have 15 datasets to replicate, that would be 30 scripts then xD.

[@兰翔-p6x]

I still hope that unraid can realize these automatic backup functions in gui like truenas

[@bluedog_fl]

Just to clarify, source/dest datasets can only be top level(?) For example appdata and not appdata/nextcloud. When I tried to backup specific programs in appdata, the backup would fail

[@Harjinder.Australia]

25:23 how is everything a child dataset of appdata? do I have to create that for individual containers?

[@cam_934]

Cheers, this is new to me but when you have made a backup to another zfs single disk how do you restore a snap shot from that backup single xfs disk if something goes wrong with the main zfs array? Backups are kind of pointless if you are left scratching your head wonder what do I do now to restore.

[@BrondRando]

There it is! Awesome. Been refreshing your channel constantly watching for this video

[@SpaceinvaderOne]

thanks for your patience.I hope you find it useful :)

[@flx420]

i made a cache pool with 2 1TB NVME as a ZFS Mirror. Question now that i have: How to replace one NVME in case of failure. Just replace the Drive and Unraid will rebuild the mirror itself?

[@floydblackbourn7232]

Thank you for finishing this multipart ZFS series for this version of UnRaid. The one week anticipated time-frame for its release came and went long ago.Your videos explain a lot, are well done and appreciated. I watched this video can see why it was delayed and realize how much work went into the creation. I can now set up the new server which has more drive slots and I can max out my drive limit. I used to do Microsoft Tech Support before it moved to India and am still learning new things. Excuse my impatience but if I was a food item my Sell By Date expired years ago and my Expiration Date is faded badly enough that it can't be read.

[@PaulShadwell]

I have set this up a while ago but after upgrading my ZFS Cache disk from 240GB to 4TB I would like to recover the ZFS datasets that I backed up. How can I perform a restore from the snapshots?

[@weilkevin]

This guide is great! I had set it 8 months ago and didn't look back. Now I'm seeing daily backups since then and I've got 6k snapshots. Any advice on troubleshooting this? I didn't change the default retention settings.

[@acrusso1]

what plugin is giving you sub tabs under the main tab? Great video/ Great script / Great job!

[@mattym00]

Did you ever figure it out? I have been hunting for the plugin too

[@mattym00]

Just found it :) Go to Settings->Display Settings and change Page view to tabbed.

[@Gex2501]

Thank you so much for this and all of your hard work. It is truly appreciated!

[@SpaceinvaderOne]

You are so welcome!

[@rgray99c]

Question, can this do recursive? I have quite a few datasets, and setting up a user script for each one was a pain, and now I've found an issue in that if one is running when the next starts it throws an error. So yeah '-r' option, if so where?

[@sabonim38]

Maybe a stupid question, but I saw your main tab is decided up in sub tabs. How to set that up?

[@xamievilx]

After updating to the 7.0.0-beta 1 i am now receiving the following error for replication: "ZFS replication failed from source: cache/appdata to disk2/zfs-backups/cache_appdata" any ideas on how to correct this?

[@genghistron1858]

Am I missing something for restoring a snapshot from the array disk? If I use rollback, I get a message saying it's successful, but nothing is restored to the cache drive. I guess I could copy all of the files out of the backup drive and move them to cache, but it's a little inconvenient. The only way I can get anything to restore is if I roll back a snapshot on the cache drive itself. Edit: maybe I'm going about it wrong, I was thinking of restoring snapshot from my array to cache, but everything is replicated in the share, I can move from backup to cache. Either way, I'll probably give it a whirl since I'm adding another NVME drive and my current + new will have to be formatted (guessing because of ZFS+encryption on current).

[@tommyjackson3811]

Yes just tried to test my backup snapshots on my external ZFS drive and get the same result as yourself. (successfully rolled back but data is not restored) Would love to know how to make use of these snapshots otherwise this is all pretty pointless

[@davidshirley3903]

What is the best method for restoring a snapshot from the backup drive?

[@olitrolli8695]

Hi Spaceinvader! Awesome video! Seems a couple of people including myself are experiencing that snapshots in the destination folder aren't being deleted to mirror the source 1:1. Any idea on a fix for this? syncoid_mode="strict-mirror" so it should work 😕

[@buster82284]

Awsome Video thank you! I have 3 Questions for you. 1) If I want to snapshot all of my shares is there a way to add that to the script or do i have to create a duplicate script for every other shares? 2) What happens if the drive has no more free space will it auto delete older snapshots or fail to create new snapshots? 3) how many hdd's would you need for say a 84tb snapshot backup? I have 6 free 14tb drives (mostly empty + a 14tb parity), and 14tb for zfs backups) but id like to have everthing snapshotted. Is it a 1:1 ratio? would i need 6 14tb zfs backups?

[@gustratcat]

Hello from greece!! i would like to point the need for shadowcopy implemented in shares, or how to do it, i tried some configs but nothing seems to work with these snaps!!!!

[@jamiedunbar3063]

Will ZFS master still be needed when Unraid completed the ZFS roll out?

[@climber455]

Awesome video Ed, thank you! Everything is working great except the script doesn't seem to be pruning the snapshots in the destination. I have 6 hourly snapshots set and those get pruned on the source dataset but not on the destination. What am I missing?

[@tunatron3280]

Noticed the same issue- Have you figured out a solution?

[@thebaldconvict]

@@tunatron3280 Same problem here, I have 13 daily snapshots instead of 7. I looked through the script and there doesn't seem to be anything in there about pruning the data other than in the source. I think something is missing either in my setup or the video.

[@JByteTX]

I'm having the same issue. Snapshots aren't getting pruned on the destination.

[@tunatron3280]

@@thebaldconvict there's a pull request on his github now that fixes this

[@tunatron3280]

@@JByteTX there's a pull request on his github now that fixes this

[@FuzzDodue]

Very useful video can't wait trying ZFS on my UNRAID.

[@alexandercedeno]

Should I stop Using appdata backup pluging after completing this?

[@rocknthesombrero1651]

Yes!!! I've been looking forward to this video.

[@SpaceinvaderOne]

Thank you for your anticipation and patience! I wanted to ensure the script was refined and polished to perfection before releasing the video. I truly hope you find it worth the wait and enjoy the content! 🚀

[@sonny8085]

Where do you get your thumbnail graphics from? i.e the deathstar with the UnRaid logo on etc?

[@heimdall_gatekeeper]

@SpaceinvaderOne: 1st thanks for your great work! You imply dataset replication to remote servers. In addition to local replication from cache to an array disk, I would also like to replicate the datasets to my backup server. How does this work? I assume, I have to somehow mounte the remote ZFS dataset to my server, but how?

[@ChuskyXX]

Isn't dangerous to snapshot database files or open files in general? A part of the content is in ram so it gets lost. Corruption can happen so I think you need to stop the containers or dump contents anyway.

[@kingpug9717]

Thanks so much for all your videos, you are such a help to the unraid community. Regarding the ZFS Master plugin do you or anyone else here know anything about the issue where all ZFS pool drives are constantly written to when this plugin is installed and the web UI is left on the main tab? I had installed it after watching the previous videos and uninstalling it was the only thing that fixed the problem, has that issue been fixed or does using the tabbed view help mitigate it?

[@immildew2]

Just set automatic refresh interval to 0 and it won’t constantly ping/refresh the drives. You can still manually refresh.

[@seriphim8542]

worked great under 6.12 now under 7 beta 1 it looks like the sanoid package needs updating. Now getting "mbuffer: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory"

[@Iam2lazy2register]

I have followed your tutorial setting up an appdata backup from my ssd to my ZFS array disk and it's working great. I have now decided to encrypt all my disks. I have watched your tutorials how to use mover and unbalance to encrypt disks one by one, but could you perhaps make a video of what to watch out for when doing this with ZFS Disks (in terms of snapshots and datasets)?

[@richardskinner6391]

How would you restore from the backup?

[@DTMHibbert]

Excellent video as always - how do you have your main page in unraid grouped into Array, Cache, etc?

[@goddroid1833]

settings-display set-page view

[@PeterCarlsson-k7n]

Thank you so much for another great video. Every time i watch a tutorial from you i always pick up great stuff besides what the tutorial is about. I'm not that skilled so i like many backup fallbacks. I have a few questions: - "mbuffer not available on source [local machine] - sync will continue without source buffering." i guess this is because i'm writing appdata from and SSD disk to a mekanic one? - Now i have appdata in two places and on both i have the ability to revert a snapshot. I don't get where i should do what and in witch scenario? - I have an GDrive mounted that i can put stuff i want to secure in the cloud. Should i run a rsync from the appdata on the SSD cache or from the mekanic zfs drive? Thank you again for your very easy to follow tutorials. /Peter (Sweden)

[@Gragorg]

Excellent video. So at the end the snapshot said 268k. Is that the actual space used to for each snapshot?

[@JM-br4ey]

Anyone know how I can replicate a zfs data set to another encrypted zfs data set? Everything works great until the target is locked

[@Anadoliev]

Thank you for the great video. I hope people understand the time and the effort needed for all the content you do. I did everything from the video then did some tests and everything is working. But i don't understand why my zfs disk on the array is always spin up even i exclude it from all of the shares? It is used only for the zfs replication. And is it possible do make the replication on an Unassigned Devices to prevent the array to spin up?

[@samuelcopelandphoto]

Thanks! For the response to my email

[@39zack]

While I was waiting for this video, I managed to setup more or less the same thing with znapzend and it has been working fine so far :) But with this method, you want to create 1 script for each dataset you want to snapshot/backup? And if you want hourly snapshots I guess you need to set the script to run on the same interval as you want to run hourly snapshots? But learned now that you can have the main page tabbed :O

[@SpaceinvaderOne]

yes each location requires a separate script and schedule

[@CptDayDreamer]

How do you got tabs on the main page? Makes it the loading faster because my main page is terribly slow.

[@goddroid1833]

settings-display set-page view

[@stayupthetree]

I do love that you choose Raw on github, when there is a copy button right next to it that will place the contents into your clipboard. That being said, thank you for all your ZFS videos!

[@deanrparry]

Awsome video as always :) I am looking how to pair SSH keys between two servers preferably over the internet? could you point out if this is documented or how to do this at all? We have 2 unraid servers at different properties and this is a dream com true ZFS snapshots remotely onto encrypted pools {dreamy} :)

[@BeastyMathew]

Can you do a video on replicating ZFS snapshots to unraid from another system like truenas ? very good content, keep up the good work.

[@frankgarcia4644]

Thank you so much for this awesome video. Makes me feel better about my cache drives data being saved since they're not protected. I am having an issue though. I started with snapshoting my appdata and replicating fine. My issue is when doing the same for my domains and same for my nextcloud folder. Any chance you can give me a clue into why? Here is the error shown in the log. *update - my issue is corrected. I believe I was having issues with my drive so I wiped and reformatted. All working now. Thank you anyway.

[@acrusso1]

Destination isn't cleaning up old snaps I have 2 weeks worth now, it's working fine on source

[@olitrolli8695]

Same problem here!

[@6n518]

Excellent video series and than you so much for the information.

[@coppertop27]

Great video! This simplifies my backups a lot! Anyone know if I use Syncthing to sync my local zfs replication to an offsite server, does all the hard linking and everything still work? I know I could use rsync in the Part B video, however, I have Syncthing already in place and wondering if anyone has does the zfs replication over syncthing? Thanks

[@drmetroyt]

New intro is lit 🔥🔥🔥 🕺🏼

[@SpaceinvaderOne]

Hey thanks. Glad you like it. Thanks for watching :)

[@drmetroyt]

@@SpaceinvaderOne please upload more videos....

[@kjcsg]

How do you restore datasets to a cache pool without the parent dataset @SpaceinvaderOne? My NVME died, but I have everything backed up with your script to a ZFS drive on my array (awesome script and video btw!). Now I'm trying to restore all the datasets to the new nvme pool with a normal tree of nvme/appdata instead of nvme/snapshots/appdata. All my snapshots are in disk3/snapshots/ (disk3/snapshots/nvme_appdata for ex). Thank you very much!

[@drmetroyt]

Sir .. please upload more videos , update the old content according to latest version of unraid its really helpful and we all thank you ❤

[@IEnjoyCreatingVideos]

Great video Ed! Thank you for always sharing with us!💖👍😎JP

[@SpaceinvaderOne]

My pleasure!!

[@IEnjoyCreatingVideos]

@@SpaceinvaderOne 😎

[@FlexibleToast]

I can't imagine thinking maintaining and rewriting your scripts over and over is easier than Sanoid and Syncoid. I think this video would be 10 minutes if you showed how to use those tools and the script would end up being two lines, one for kicking off sanoid and one to kick off syncoid. If you don't have a zfs target for the backup, borgmatic is another easy way to write a config and just let it handle it.

[@DanPennington]

Awesome tutorial! Thank you!

[@SpaceinvaderOne]

You're very welcome!

[@ozlazarsky]

Thanks Ed, You are an ace😊

[@SpaceinvaderOne]

Your welcome. Thanks for your kind comment :)

[@ozlazarsky]

@@SpaceinvaderOne seriously. I really don't know what I would do without you :)

[@glasshalfempty1984]

2:45 If I knew how to do that then yes...I might have by now :P

[@enkrypt3d]

fantastic video! thank you

[@kumper33]

Yes! Thank you

[@SpaceinvaderOne]

You are so welcome!

[@Nightowl80]

Love the videos, thank you for all of your hard work. Would love to see how you did the “Plex” zfs pool, does it refer back to your Unraid server media folder??

[@ChrisPlough]

Thank you!!

[@wojtek-33]

Weird that you would gloss over live data replication as it is ALWAYS risky, especially if any database files are in use. This is the reason Proxmox gives you the option to freeze, suspend or stop a VM/Container before backing up, with freezing even being deemed a risk.

[@benjmuller1523]

Wow! 😍

[@SheliaHicks-y7w]

Jackson Maria Taylor Susan Martinez Michelle

[@Redneckrampage]

I like unRaid because I can have different size drives. If I convert to ZFS, will I need to have all the drives the same size?

[@39zack]

Not in the array, but it does not make sense to have all drives zfs in array. Its messy when the datasets gets split over different drives.

[@drmetroyt]

​@@39zackthe ONLY advantage of keeping all disks in array as zfs is snapshots ... The other benefits of zfs are none ... So it's better to keep one or 2 disk as zfs in array .. 2nd thing is if all disks are kept in zfs than when we access data from one drive or write to that disk all disk will spin up .. consuming energy ... Its better to make a separate zfs pool and use and keep 1 or 2 disk in zfs in array rather than all

[@39zack]

@@drmetroyt yup, thats how I run it now after trying zfs for all array drives. Sure, snapshots are nice, but the restore got messy when a dataset is split over multiply drives and you ether need to restore all for all drives or figure out where the data you want to restore is/was

[@39zack]

@@drmetroyt forgot to mention. The one drive I have in array that is zfs is also removed from global shares so nothing gets written to it by unraid. I use it purely for zfs replication backup and its in the array so it gets added to the parity :-)

[@soundmedia222]

There it is! Awesome. Been refreshing your channel constantly watching for this video!!

[@SpaceinvaderOne]

Thanks for your patience :)