Backing up a Linux Server with rsync

[@mikescott58]

I have used rsync for several years and love it. I also use the "poor man's data deduplication" trick. Each backup starts with copying the previous directory locally, but as hard links. Then each backup is effectively a delta. If a file hasn't changed, it only takes up a directory entry and is a pointer to the data. The OS takes care of all the housekeeping issues.

[@180doman]

Thats really creative idea but also a double edge sword because it means you have only 1 copy of your data. If anything happens to your backup files then you're pretty much screwed. And overall backup is about making independent copies.

[@bertnijhof5413]

Nice explanation of rsync. I used rsync a few year ago, but since I use ZFS, I only use rsync in combination with KDE-Connect and my phone the Samsung Galaxy S5 and its 64 GB Micro-SD-Card. I backup in both direction to keep desktop and phone data synchronized. My 2014 Samsung S5 is basically my off-line backup for my private stuff (photos, family videos, music and documents) :) :)

[@mtbdude641]

This is awesome can you make a more in depth video where you schedule it with a chronjob and setup the deduplication you talked about towards the end of the video

[@olehharhat3070]

How about running Ansible role with AWX, you can schedule it and see logs etc etc.. :)

[@dirkpostma2135]

Hi Jay, great video again, thx. Since shortly I use restic to backup, its special and awesome.

[@safiqulalam9587]

Your tutorial is neat and clear for everyone no matter beginners or advanced user . Awesome 👍

[@manettteboh3053]

this was a really nice tutorial. rsync is the best server backup tool and I use it a lot.

[@jeffherdz]

Great video as always, Jay.

[@DL-xf3ur]

Great as always. Making things clear for us one video at a time. Thank you.

[@benjaminwharton6264]

This is one of my favorite channels

[@lawrencedoliveiro9104]

2:25 Worth pointing out that the device names are “/dev/sda” and “/dev/sdc”, and not something like “/dev/sda1” or “/dev/sdc1”. In other words, he has the entire disk formatted as a filesystem volume, instead of setting up a partition table (even with just one partition) and formatting the partitions. Linux lets you do that. Does your favourite OS let you do that?

[@leonbishop7404]

windows doesnt recognize those as volumes and suggests to format them 💀, lol

[@AquariusTurtle]

Great tutorial. The next logical step is to create a public/private key, install the public key, then setup the transfer over the internet using SSH. If you don't specify a passphrase,, then it becomes possible to fully automate this using a shell script. I'm new to Linux but even I figured it out because of instructions on various websites.

[@AlbusRegis]

If you are going to sync stuff through the internet you actually want to use a VPN, because chances are it is not going to be the only service that is going to be running around between sites. Using a VPN allows you to have only one exposed port to the internet, which makes it a lot easier to filter for you and harder for an intruder to get information from it, since no matter how may different services you pass through it or the changes you make they are not easily observed from the outside.

[@AquariusTurtle]

@@AlbusRegis I know nothing about Linux, but I can follow excellent instructions on sites like DigitalOcean (most Linux instructions are otherwise beyond terrible). I totally agree with you about VPN but how do you set up VPN so that only a few things go through it between two Linux machines? The source computer runs PHP/Apache and is a front facing website by design. But backups/syncs would be the only thing that would need to go through VPN. I assume Linux would let me direct stuff through VPN by targeting rsync to \mnt\VPN or something like that? How do you actually set up VPN between two computers?

[@lawrencedoliveiro9104]

SSH is only one port, and you can tunnel a lot of services through that.

[@BhekizweShongwe]

Awesome stuff. Thanks. I needed this.

[@mkintzel]

Thanks for the video; I watch most of your videos and have learned a lot; thank you. I would love to see you go a bit more in depth as that is part of the practical application of learning Linux. The syntax to move files with rsync is great but how to really package it up as a viable scripted run out of cron solution would be better. For instance, I tried to figure out how to really design a back solution around rsync and finally gave up and found borg and used that, but you mentioned that in the video that dedup and other topics (assume incrementals and/or differential, compression, etc.) were beyond the scope. I would love to expand my knowledge and learn more about a complete solution built around rsync. Thanks again

[@dingokidneys]

That's a good simple mechanism for creating backups. I use something similar myself. Just some other things I've been pondering: I want to keep backups unavailable to any hacker that gains access to my internet facing machine and how to minimise file duplication in the first place. I think that if you run the backup from the backup server and make sure that the other machine doesn't have credentials lying around to get into the backup server, you could be more secure. In fact you could set firewall rules to lock everyone out of the backup server and have it open a port when it goes to run the backup. Next, rsync has a mechanism for creating hard links to unchanged files instead of copying the unchanged file. See '--link-dest=DIR' in the rsync man page. Timeshift uses this method with rsync on non BtrFS file systems to avoid duplicating unchanged files. Undoubtedly, you can get very complex very fast when you look into what rsync can do but it's worth having a look to see how you might gradually improve your backup mechanisms to achieve better and more secure backups.

[@cjwworld]

I really appreciate your method of educating us. You have a good way of speaking and I have learned a lot from many of your videos. In fact, your videos is the only reason, I have been successful in getting Nextcloud running on my own server. I love the simple and cool things you helped us to learn. Very pleased

[@eyes1609]

Hello, great demonstration. How to add a line to delete old backups? THANKS

[@aaamos16]

Can it update changes only?

[@JohanEkelund84]

Great as always! Im not too fond of the music in the background though, it gets kind of distracting.

[@alainmuzik]

Enjoyable great and clear explanations, keep going

[@evanathea]

I wish you would explain the options -avz and +%F better

[@elements88xyz]

man rsync or rsync --help - will show you all the details on the command's options.

[@lawrencedoliveiro9104]

2:40 I normally delete that “lost+found” directory. If an fsck run ever finds anything to put in it, it will automatically recreate it.

[@wildmanjeff42]

Thanks for the video Jay, I love command line videos and usage, even though I am still a newbie to it !

[@toweliethetowel8280]

Maybe just because you're a newbie, like me :)

[@clivewi9103]

An interesting video, a little bit more information on what the command line switches do would be helpful.

[@zauliuz]

Thx for your tutorials. They are really helpful. :)

[@fredtheilig9636]

Very good. I use the -h (human readable) option so that the output numbers are a little easier for my brain to process and had not considered the -z (compress) option. I'm curious if it improves the backup speed. I just keep one backup and use --DELETE to essentially make my backup an exact copy of my home folder at the time of backup. I may convert to your method. Also, the --EXCLUDE-FROM= option is very useful for excluding junk from the .gnome, .local, .cache, and similar directories.

[@d00dEEE]

Speed with -z probably depends a lot on the both the CPU available to do compression and line speed+traffic. If you have an ARM CPU running over a local 10Gbps backbone to your server, I'll bet it's slower, but with a decent CPU and 1G (or even worse, a home internet connection), then I'd put money -z being faster.

[@SergiRodriguesRius]

@@d00dEEE conclusion? always make a couple of tests comparing both alternatives 🙂

[@FossWolf]

using date is great idea

[@МартинРоджев]

Extremely useful, thank you

[@aaronperl]

Thanks for this. I've "known about" rsync for a long time but I've never gotten the hang of using it. For some reason I always thought it needed a server process as well. Must have been a misinterpretation when I tried reading the manual many years ago.

[@AlbusRegis]

You can have a server process for it so that you can automate remote syncs that are more complex that what is shown in the video.

[@jim7smith]

Nicely done, Jay..... However, like many tutorials on backup, not much mention is made of the actual recovery procedure if you have a source drive failure or operator error causing loss of data. Do you have a video should how to do that from the ending point of this video?

[@jeytis72]

How can we restore data back to the same folder if needed? Thanks

[@VulcanOnWheels]

8:56 Couldn't you have used $(date) without first putting it into a variable?

[@humming25]

this video is really very helpful. Can you please make a video on how to backup data on a linux server from other linux server or in same server but in different folder using Rclone and automate it?

[@yomajo]

Would love explanation of difference between rsync command and rsync service.

[@worthmind]

Please make a video about turn server audio video calling system setup for nextcloud

[@omaralhalboosi2713]

This is beyond just liking or commenting on your videos , thank you so much. I wonder whats gonna happen if just use DATE=(date) without what you added after date

[@clivewi9103]

Suggestion for another video, how to swop a Sata disk for a SSD while taking all of the installation and user data with you.

[@alfuller9809]

Thanks for the video. Can you refer me/us to directions on how to use TrueNAS Scale to back up Windows hosts?

[@steves9250]

Is there any guidance on which directories should be backed up? Should you start with / or only selected directories?

[@jonathanrider4417]

Another fine video Jay - thanks! Hope you lose the background music in future - I believe I am not the only viewer who finds it extremely distracting if you are talking over it.

[@mathieuleclerc4136]

Does it works with a raspberry pi?

[@dtmbmw325i]

I’d like to see scheduling and deduplication. What about encryption?

[@sagargandhi8609]

Hi, using Rsync the files are getting transferred in the order from the oldest to the newest, is it possible to change the order in which they are being transferred?

[@greatosazuwa1656]

Hello i want to backup my entire server both with the configurations and settings like apache2, mysql, hypercorn locally so i can reuse it when i want to How can it be done?

[@KhanAmmar-i3o]

HI @linux academy can you make a video on to implement backup procedure on FreeIPA servers

[@KeithDavey2014]

I have a problem with using just RSYNC as a backup solution. rsync does replication of data. It has no versioning. A good backup solution needs to be able to store more than one version of a file. Be that different versions of the file stored in full, diff, incremental backups, or a backup solution that is versioning aware. borg-backup is a much better solution. Archives can be automatically retained based on recovery point needs, archives are deduplicated and optionally encrypted. It also can audit and validate its own archives.

[@alain-pierrep.312]

Borg can also mount restore points and prune on demand. IE : "keep 4 dailies, 8 weeklies, 10 montlies and all yearlies". All in all, Borg takes a little bit to set up but is *awesome* (and, IMO, checks all the boxes of a good remote backup solution). :)

[@lawrencedoliveiro9104]

rsync can do versioning. I set up a system for doing exactly that for a client. It kept two backups a day going back about a week. Each was done incrementally, but thanks to the deduping magic of --link-dest, each snapshot looked like a full backup for restoration purposes.

[@suryateja2524]

Please do the videos on linux server (Ubuntu) adminstration.

[@АнгелИнокентий]

Thank you, but how make a automatic backup?

[@jjb7920]

Put the rsync command line in a bash script file, and use a cron job to run the script on a schedule. Check out Jay's other videos. He covers all the steps. I just watched them myself.

[@jamj2000]

Hello Jay. What about a video on how to backup using git ??

[@samuelaponte4225]

Can you teach how to make a iso backup of the current os??

[@samuelaponte4225]

Me hubiera gustado que hace años alguien me hubiera enseñado

[@donaldwilliams6821]

Nice video. I hope you do other videos on rsync as well. I also use the mirror function of rsync to sync two NFS ISO servers. Maybe showing how to use SSH keys so you don't have to enter your password each time the script is run ? There is so much you can do with rsync. Thanks again.

[@elements88xyz]

you can set up SSH key authentication ;)😁

[@barfnelson5967]

can you do one of these on using xargs to parallelize the backup for those of us with large amounts of files?

[@AlbusRegis]

You do no need xargs, rsync can pull from multiple directories if you use the right arguments, it is in its manpage

[@send2gl]

Always got a bit confused with backing up some dynamic system files, that has generally put me off of rsync. Nice video though, well explained.

[@AaronWeissRocks]

Can you please do a follow-up to show how one might restore a backup file?

[@AlbusRegis]

You just rsync the other way since rsync just makes a normal directory structure on the destination, do not forget to sync the permissions though.

[@raul230285]

Thank you very much for your videos, you are the best. Could you please make a video of the VPN Nebula. thank you

[@ZaG-yo3fd]

I tried running DATE=(date) It didn't work. I want to see the files with both date and time. I thought it would be as simple as leaving off the '%F" and now I'm all confused. I thought you can pass the command date into a subshell equal to DATE and just run it. Any help on why I'm not able to do this without the formatting you used? Thanks.

[@TradeMill-i1n]

This an awesome tutorial! Also, I really like the background music❤. What's the title of the music if you wouldn't mind? Thanks!

[@harunaadoga]

Thank you!

[@leonbishop7404]

good tutorial!

[@kapiell2999]

its really helpful

[@sebastianmoustakas2094]

I have an VM with Linux Ubuntu Server 16.04 with some running services and i need to update the O/S. The VM has two disks . dev/vda with the O/S and another disk dev/vdb for backups. Is it a good idea to use the rsync command to backup the whole vda disk to a folder on the vdb disk before upgrading the O/S? I want if something doesn't work after upgrading to restore to the previous state.

[@adjusted-bunny]

For my home folder I use rsync and for my system folders I use rsnapshot (which is also rsync).

[@evgeniyvalchev5317]

RSYNC with ZFS ?

[@gabrielgodoy7846]

You are a genius!!!! Thanks 👍

[@lawrencedoliveiro9104]

4:05 Either use “sudo” or “su”, no need to use both.

[@arminmansouri1743]

Wouldn't it be better to run the rsync from the backup server instead of the host machine. This will be more organized and cleaner if you have several machines to take backups from.

[@fredneedle123]

Can you use chrontabs to automate the process and how would you do this without it asking for the password for the destination directory?

[@maginos1310]

Yes that’s possible. In the crontab of the user root (sudo crontab -e) specify the time when you want the script to run and put the command bash /path/to/the/script . For passwordless authentication I recommend to look at private/public key authentication. I hope this helps you.

[@AlbusRegis]

@@maginos1310 It is also possible to set up a rsync daemon to achieve the same and it is not too difficult

[@lawrencedoliveiro9104]

Using SSH as the transport is probably the easiest. You likely need to set that up for remote management purposes anyway, so why not minimize the potential remote attack surface.

[@andrianrahardja8854]

Really love your video, but can i request about how the best practice cleaning all logs maybe per weeks or per days using cronjob? Could you please make a tutorial about it. Thank you

[@mihai6564]

thanks

[@lezz27]

Thank you for posting this. I enjoyed the short session. I do have a question. My environment includes Windows and Linux machines. I have a SMB file share on Windows machine which is accessible via username/password on my local LAN. Would you let me know if I can use Rsync on Linux machine to backup data from the Linux box to Windows SMB share? If yes, how? What command do I need to use? Thanks in advance.

[@lokideus5094]

First, create your mountpoint: mkdir /mnt/windowsshare Second, mount your smb-share: sudo mount -t cifs -o username=,password= //windowsshareIP/ /mnt/windowsshare Thirdly, rsync it: rsync -avz /mnt/windowsshare /path/to/your/backup/location/on/linux To make all of this run automatically, you can edit your fstab to mount on boot and add your rsync-line to your crontab.

[@LearnLinuxTV]

Thanks for posting that! To add another detail, you might get some errors pertaining to permissions when you copy files to a Windows machine. So long as your data makes it to the destination, you can ignore those warnings.

[@lawrencedoliveiro9104]

Doing bulk copies to/from a file server will likely be very slow. A faster way would be to shut down the Windows server and boot it up with a Linux distro like SystemRescue, so that one end of rsync can run on that and access the files as a local volume.

[@Viking8888]

Hi Jay, I really have enjoyed learning more about the linux command line from your videos. I do have a question for you though. I am wanting to learn more about bash scripting. In particular, backing up an Ubuntu server instance and restoring folder/files from my NAS to the Ubuntu server. The scenario is this. My Ubuntu server has many users... user01, user02 and so on. Each user has the same folder with important files in it. The folder is called Default. How can I automate backing up the Default folder for each user and subsequently restore the Default folder to those users if something goes catastrophically wrong with the server and it needs to be rebuilt. I want to do a daily backup of each users Default folder but only have what's changed, copied to the NAS. I read rsync is good for this. Can you or perhaps someone else help me with this? Thx.

[@ElegantSolutions]

I appreciate that your newest videos do NOT use this distracting background music.

[@michaelrotter8561]

I like it!

[@astratheus]

I use rsync on Windows!

[@dingokidneys]

That's really good. Windows needs all the help from open source tools it can get. :)

[@TheROck3052]

Please do not use rm -rf * even if it is a test system.

[@paulstaf]

You didn't explain the real reason why someone would want to use rsync, in that it can save on bandwidth and time by backing up only files that have changed..... nor did you explain any of the options you used.

[@darthkielbasa]

I was unaware of the power of rsync or the two points @paulstaf made. I’m going to poke around with this utility.

[@angelozou3914]

平衡，旅。😃 老师会中文吗？

[@Mrsbencharmcaster]

Should be start with lil basic.

[@kthfriend]

If we are using rsync over the Web won't we need port forwarding? And what port is normally used? And what about encryption as well?

[@gadflyofhumanity_6847]

Firstly you NEED port forwarding, a vpn, and you'll need to compress and encrypt the payload (i.e. the backup).

[@lokideus5094]

Rsync uses port 22. Of course you need to be able to establish a connection between the two locations to be able to sync. If you want to encrypt the traffic (which would be quite smart if you are doing it over the internet) then you should look into solutions like duplicity or tartarus.

[@LearnLinuxTV]

It depends on which side is publicly exposed to the Internet, if any. If you're using a VPS instance, then you shouldn't need port forwarding if the rsync direction is out of your home network, and towards the VPS instance. The reverse of that would need port forwarding.

[@dingokidneys]

The mechanism demonstrated uses ssh as the underlying transport method which gives you encryption by default and uses a default port 22. This also means that if you can reach one end from the other with ssh, you're good to go. As Jay mentioned in his reply to you, home to a VPS should be a breeze. The other way around brings more complexity and some risk. Jay's demo showed driving the backup from the machine you wanted to back up, but you could also do it the other way around; rsync doesn't care, so it doesn't matter if you're wanting to back up a VPS to your local machine, or your local machine to a VPS; you can do it. Of course if you're wanting to back up your dad's home PC to your machine, there will be port forwarding and possibly dynamic DNS to handle as well as the software configuration. It's not actually that hard but does require a bit of planning and knowledge.

[@webalizer_yt]

The title of this video is a bit misleading as you backup just one directory not the entire server or the servers system files.

[@lawrencedoliveiro9104]

If that one directory is “/”, you get the whole server.

[@md.redwanhossain8822]

Your old video style was perfect. Giving music within a serious learning video doesn't fit well. Consider using your old style again.

[@vilecoyote5673]

Please PLEASE stop using the heavy variable volume background beat, it does nothing but disrupt any concentration on the knowledge you are trying to impart. It's not required and the free silent periods allow the message to be grounded.

[@JosephWilliams-v9x]

Hall Jose Clark Sandra Johnson Gary

[@RayZde]

not really backing up a server. it's backing up files to another server.

[@rursus8354]

I hate the music and the sound effects. I'll skip this video and take a look at man rsync and Google instead.

[@evodefense]

Thank you!

[@jamj2000]

Hello Jay. What about a video on how to backup using git ??

[@dingokidneys]

While you could use git to back some stuff up, particularly config files in a $HOME directory as a good example, it is not a good generic backup solution. There are other videos on KZbin describing how to use a git bare repository to manage Linux config files which are great but once again this is a very specific use case.