Get Docker organized for easier backups & replication. Trust me, an hour can save you days!

[@happysprollie]

This is why I like docker-compose, because it's easy to recreate containers and is practically self-documenting

[@AwesomeOpenSource]

Absolutely! Compose is a tremendous tool.

[@default_youtube_profile]

But docker-compose is not docker swarm

[@froid_san]

As someone still learning the in's and out's of docker, this is a very great tip. Luckily since I'm still a newbie, I've been doing all things docker using docker compose and always saving it as I'm afraid to mess up. Now I must keep in mind the data files of the containers to backup. Great vid!

[@AwesomeOpenSource]

Thank you, and it sounds like you’re on the right track.

[@microcisco]

Great channel and love your videos! I know most people don't have the infrastructure to be able to run a hyper-converged setup, but if you have 3 physical machines and SSD disks to run a Ceph storage (storage as a service - StaaS) on 10Gb interfaces. I run a Proxmox 3 physical server setup with the 3 in a cluster and Ceph Storage to form the HCI. I run the Proxmox backups nightly and it has saved my ass a few times and easily restore the backups to any of the 3 in the cluster with all attached to the Ceph storage. If you can setup the HCI environment it will give you peace of mind and the functionality to easily restore failed VM's and containers. (As a side note, I used to run LXC containers but I find it much easier to run VM's with docker on those VM's.)

[@AwesomeOpenSource]

Great information and advice. Thanks for sharing.

[@MrNoBSgiven]

Great topic. 1. Backups is a must. Snapshots of VM would have helped you recover in no time at all 2. . Portainer has a backup function,mwhich backs up the entire setup and allows recovery to a new machine in no time at all. Finally, have you tought of automation of the backup and recovery/seting up using terraform (proxmox VMs) and ansible for the "fresh rebuild" of your VM environments? THESE WITH distributed file system storage could help. I am in the process of setting something like that. I am thinking about using docker swarm to take away the headache of manual splitting of docker apps.

[@AwesomeOpenSource]

Agree 100% on the backups. And i do, now, have a backup of my VM, and it's automated as well. The docker part is an interesting one. I did find the Protainer backup (after the fact), but will start using it too...yay! As for Swarm, I'm not there yet, but was also thinking this would be a great reason to learn and start using kubernetes at some point. As it would have provided the no-down-time reliance I might want some day, while I'm re-building one server if needed.

[@gravyflex]

I'm the middle of setting up a docker swarm built with terraform and Ansible. Working on NFS persistent storage with a docker plugins. Right now I am figuring out traefik reverse proxy.

[@MmMm-oh5ct]

Its Awesome, I am on my way to Setup kind of something with ansible at the moment. Great Video

[@AwesomeOpenSource]

Thank you.

[@JPEaglesandKatz]

Also the way I've set it up is bind mount every single data folder from any container I use to a cental data folder... I backup this folder with borg backup and also create periodic toal VM backups, including syncing them to an external drive as oposed to only my zfs pool.

[@AwesomeOpenSource]

Nice. Great tip!

[@JPEaglesandKatz]

checkout borg backup.. its great.. easy to put into a cron job, backup up root owned folders, incrementalized backups, compression, protection, etc... set and forget once setup correctly

[@AwesomeOpenSource]

I've looked at Borg a few times, just haven't tried it yet. It's on my list though.

[@JPEaglesandKatz]

@@AwesomeOpenSource Cool... Yeh it is pretty good... Keep on rocking!! I love your channel, came across some really cool pieces of software that way... I was a linux noob 8 months ago... now it is starting to feel more like home than windows!

[@rockettpc2]

Great info, funny, I kinda intuitively started like this, but I find myself doing the same in my individual devices, so it was kinda the norm for me. But this is very important to do, stay organized and have backups etc. I am playing with docker swarm on a 6 Node Odroid C2 cluster and I'm using NFS for the data, I really need to build a good NAS ideally more than one for redundancy of course, but it's all allot of fun self hosting and pushing the edge of home computing.

[@AwesomeOpenSource]

Keep it going, and glad you are already on track with organization!

[@artedwards717]

Hey brain awesome I wish you could make a drive with that all on it I am learning wow on your videos

[@AwesomeOpenSource]

What do you mean by a drive?

[@artedwards717]

@@AwesomeOpenSource oh brain I mean like a CD to load on my main computer

[@CaptZenPetabyte]

Great idea as Im about to start a new project & setting it up well like this from scratch will be so much better. Could prob even write a bash.sh to copy & bring everything back up again to automate

[@AwesomeOpenSource]

Better to start off organized, but worth going back and re-organizing as well!

[@kronos3428]

Thanks for the tutorial. This has always been a big concern of mine, I like to do things just once. I recently started using TrueNas Scale which is debian based and has docker support and a very interesting thing I found out is that you can change docker root folder, so I have this disk that when attaching it to any other instance, the whole date the containers are there and also the containers, they even start automatically (following the rules previously assigned).

[@AwesomeOpenSource]

Nice setup.

[@majidkhoram2451]

Did you know you are brilliant? Or should I mention it?😉👍🏻

[@AwesomeOpenSource]

I appreciate the compliment, thank you!

[@kevinhughes9801]

Great stuff thanks

[@AwesomeOpenSource]

My pleasure

[@50_Pence]

Did that mariadb backup work OK? Or did it corrupt? Did you have to stop the container first?

[@AwesomeOpenSource]

If you have a system that writes to MariaDB a lot, then it is better to stop the container, then do the backup, and start it back up. I've been lucky so far, and my containers are generally low usage, so when the automatic backups run, no one in my home is up and using anything.

[@SaroxDuPoney]

Hello thanks for your tips. What is your container for your dashboard @16:07. I like it a lot.

[@AwesomeOpenSource]

Currently I'm using Homer, but just recorded a new video on Dashy, and I'm thinking about switching to it. I need to test it for another week or so before I decide. Video coming soon!

[@SaroxDuPoney]

@@AwesomeOpenSource ok thanks. I will be looking for your next video. Thank you very much for all your info.

[@markstanchin1692]

Ok i’m in trouble. So I installed a bunch of docker containers using different methods portainer, docker compose etc. When I ls under my home directory I only see two dirs , Heimdall and guacamole soo I thought when you installed a docker container it installed and created the Paths like in your example automatically. There installed and working perfectly but I have no idea where the dockers Containers and data is installed by default. So without redoing like 15 containers is there a way I can find out we organize and regroup the way that you’re showing in this tutorial? Thanks

[@AwesomeOpenSource]

When I look at my file system, I find Portainer stuff under /portainer/Files/appdata/ , so you may start there. There is also stuff under /data/compose. You can use the Portainer UI to backup your docker stuff as well. But if you want to organize it, you'll need to move stuff from the file paths above, and then start cleaning it up. Do one at a time, take your time, and make sure each app comes up as you expect.

[@markstanchin1692]

@@AwesomeOpenSource Thanks! Are you going to create a video on how you actually do this and how you back up your folders zip up script etc? Also I was thinking of moving data from the containers over to a Nas device not sure if it makes sense or how it would work have you ever done anything like this? Also off-topic I host a couple of websites not WordPress on site ground. Well the renewal is coming up and it’s four times my original price. So I’m thinking of moving to self hosting my sites but I like the way site ground works with the control panel. Is there anything that can be installed on a Ubuntu vm server that can act as a control portal like cPanel? I’m thinking of creating a separate VM just to host my websites does that make sense?

[@lezz27]

Nice video. I just had a crash and I know the pain in getting everything up and running. I have a mic of Windows and Linux machine. My Docker stuff runs on Ubuntu 20.04 where as I have shares on my Windows machine. I was wondering whats the best way to backup the docker files onto the Windows share. You mentioned something abt the script you use to zip and upload to a share. May you please share the script? I tried using Duplicati and Duplicacy on my Windows machine and then mounted the Docker volume on Windows machine using "Mountain Duck" application. The issue that I am facing is that there are some parts of the Docker folder that I am not able to backup using these backup apps due to permission issue. I am not sure why I have that. I tried to grant my local user on Ubuntu a root access, that didn't work either. Any help/suggestion in this matter will be appreciated. Note: All my shared folders are on my Windows machine.

[@AwesomeOpenSource]

Couple of things to look into. 1. make sure your user on ubuntu is part of the docker group. sudo usermod -aG docker {$USER} 2. Setup a Samba share, and make sure you mount it on your Ubuntu server. 3. Now you can zip your docker folder ( you may have to use sudo ), then copy that folder to your mounted Samba share. When doing this, you'll want to make sure and delete versions over time so you don't fill up your drive on the Ubuntu machine. The script i used, 1. created a gzip file of my docker folder, then 2. rsync'd the file over to my backup server, and then 3. deleted the gzip file. So each day the files were zipped, backed up, and then the zip was removed. It was a very simple script, but effective.

[@lezz27]

@@AwesomeOpenSource This is the quickest response I have every seen in YT. Thank you! I am bad with writing scripts. Would you mind sharing your script? I did setup my shares on windows with required permissions and currently trying to figure out how to mount the Windows SMB shares on Ubuntu server.

[@markstanchin1692]

Hi, I just ran your latest script, how cool! I’m rebuilding my system in a new VM and docker etc. trying to get organized the way you show in this video. The script helped. A Couple of things I noticed it does not create a docker folder for the containers it installs. Also it says it installed Portainer agent but it’s not listed under the containers with port 9001. Also the stacks that are created say they have Limited management under Portainer. Any thoughts? Will you be adding other applications to the script? Also when I click on a port from a container it takes me 0.0.0 local host I remember somewhere how to set this to the IP but I can’t find it. Thanks ! now I’m back to work .

[@AwesomeOpenSource]

Yeah, I need to check for or create the Docker folder, not am not doing it today. Not sure why the Portainer container isn’t shown if it installed properly. The IP is set from your main Portainer Home, then I believe under Environments. I’m considering more options for containers, but at some point I’m just duplicating what Portainer can do in the UI. As for the limited management, I thinks that’s what it always says for Docker-compose created outside of the Portainer UI.

[@dimitristsoutsouras2712]

10.45 Pleaseeeeeeeeeeeeeeeeee pass the link of that procedure. If it is what I think, I am looking for this for months. My use case scenario is..... in a win2019 vm which acts as n A.D and have been installed on a raid10 zfs storage (storage derived from the same server the VM is installed)I have attached an extra portion of another raid10 zfs (derived from the same server as well).....what I left thinking afterwars is... if I migrate the VM what happens with that extra disk I have attached to that VM since it is not a shared folder or iscsi... it was just a block device that windows format and used. I m stuck there for ages!!!

[@AwesomeOpenSource]

Instead of a script, I've found a great container called Duplicati and just recorded a video on it, as well as how to use it to backup your docker folders (or anything else) on a nightly schedule. It's super great. Coming soon!

[@dimitristsoutsouras2712]

@@AwesomeOpenSource I already know about Duplicati and UrBackup and other programs as well. I have to wait for your video and see if it's use case scenario can be applied to me too. PS By the way for me it would be useful a program that can take a block level backup and not file level.

[@AwesomeOpenSource]

@@dimitristsoutsouras2712 Timeshift on Linux can do Rsync backups / snapshots, and does diffs between the files, instead of complete files. Not block level exactly, but may help. I believe clonezilla can do block level, but you have to boot into Clonezilla to do it, or run it as a VM.

[@dimitristsoutsouras2712]

@@AwesomeOpenSource Thank you for your answers. ..... and that is exactly what I do. I use rescuezilla (even better than Clonezilla) and backup each disk (it is a mirror), do the upgrades or whatever I need to and then if anything goes very wrong, I revert back. Problem is (as you mentioned) I have to shutdown the server boot from usb stick make 2 backups (25min each on a 120gb disk - for the OS) and takes time for a production system.

[@AwesomeOpenSource]

@@dimitristsoutsouras2712 is this for Windows or Linux?