Automatic Container Updates - Watchtower Docker GitOps

[@NightHawkATL]

I have been using Watchtower for a few years as well with Gotify notifications. I was updating my containers and stacks manually for a year while learning docker. Once I got up to where I am now with multiple docker instances to split up workloads and on different physical hosts to minimize downtime, I had to implement Watchtower. I have almost 100 containers so updating would take forever. I use Commafeed to pull RSS/ATOM notifications of new release notifications from GitHub pages to know if there were any breaking changes after updates that I will need to fix. I also utilize Monocker (MONitor dOCKER) to monitor the containers to let me know when they are rebooted or are in a boot loop or stuck in some way after updating automatically. I do have a few containers that don't get updated automatically so I can control the updates that may have breaking changes on each update (Tube Archivist).

[@StephenCunningham1]

Did you have manually set up every RSS feed for each container

[@cyberjohn44]

Great video. Been using Watchtower for a few years now and make life easier when running a home-lab.

[@JamesTenniswood]

Been using this for a few years, with pushover notifications, but still learnt a few new tricks for the video 🎉

[@bluesquadron593]

One cool use for the API call I found that the watchtower is only being called when I am (my phone) detected to be at home in Home assistant. No updates when I am away, in case something goes ape.

[@RaidOwl]

Yeah I was thinking of setting mine to Monitor Only mode so I’m alerted of updates then when I’m ready I just hit the api.

[@theWSt]

That's very helpful, thx for the video! 👍

[@TheDefaultgameer]

I started using this a few weeks ago, was fine until suddenly my link to heimdall stopped working. Looks like it kept changing the port on that and I believe another container as well. Otherwise I loved having something to help manage this.

[@RaidOwl]

Hmmm strange. I’ll continue to monitor my setup

[@bugdozer314]

Watchtower is great, but I do not let it run automatically. Run it weekly or so, observe behavior, I don't want to be surprised if it breaks something by grabbing a docker image that an upstream broke. Also, be aware, watchtower stops all, then restarts all. This means your containers are down for that interval. If you have linked containers, this is even worse, as they'll be turned down in some order, then brought back in some order, possibly with waits for health check successes, actual pulls (downloads), et al. (And it actually performs worse at times than you'd expect, many minutes of downtime when nothing seemingly would require it.) So I've taken to running two passes with watchtower, one that only specifically targets linked containers, and a second pass which specifically does any non-linked containers. This is done via some "interesting" docker inspect calls before calling watchtower. It'd be nice if watchtower made this an option directly. EDIT: typo, clarity.

[@chrislowe8085]

so my Server runs Unraid I have a plugin called CA Auto Update Applications it checks on a set time or day for updated dockers and plugins on my box and auto updates them and removes the old image of the docker, kind of does the same thing watchtower does. i love this plugin i don't have to do the checks :)

[@bluesquadron593]

@6:15 watch the lamp!

[@ystebadvonschlegel3295]

I’ve seen so many people say watchtower broke their system by doing some update that had breaking changes or major update in config files and then borked their system. That’s why I haven’t started using it. I wish there was a way that developers and update containers like docker could say “hey, don’t update until you change some things” to keep this from happening.

[@stephenreaves3205]

That's what tags are for

[@ystebadvonschlegel3295]

@@stephenreaves3205 Sorry I’m not understanding - is there an actual tag that is standard for that? I missed that part. If I could have this update everything except breaking changes or updates that require new parameters for the container I would install today!

[@stephenreaves3205]

@@ystebadvonschlegel3295 so it's the combination of semantic versioning and tags, but ultimately it's still up to the developers discretion. Typically, most people use the 'latest' tag which makes no expectation of stability. The developer could also publish tags like 'v1', 'v1.1', and 'v2'. The v1 tag could be a pointer to the v1.1 tag and you can base your containers on the v1 tag. Then when they add a v1.2 tag and update the v1 tag to point to that, your container gets updated and (according to SemVer) it should be compatible. Any changes that are not compatible with v1 should go into the v2 tag. Some people simplify this by having a smaller list of tags like 'latest', 'stable', and something like 'next'

[@7MBoosted]

I have definitely been using watchtower in my docker environment. I will say though, that I dont do any of the fancy stuff you do, I just let it run for all my containers, take periodic backups of my portainer instance to revert if anything goes wrong. I haven't ran into any issues over the past 18 months or so using it, but the devs do disclaimer that watchtower is not best practice, I don't care though.

[@RaidOwl]

Yeah I have daily backups of my entire docker instance so I’m fine with it

[@ystebadvonschlegel3295]

Do you have portainers backups automated or do you do this manually? Not sure how you do this.

[@7MBoosted]

@ystebadvonschlegel3295 I bet there is a way to do periodic backups, it may be a business version feature though. I just create a full backup every time I log into portainer to clear out the old unused images.

[@RaidOwl]

@@ystebadvonschlegel3295 Automatically since its a VM backup in Proxmox

[@balsman225]

Thanks for the video - good stuff. I decided to disable the schedule and just use the HTTP API and setup a button in HomeAssistant to update all the container when I want so I'm not caught off guard if an update goes bad.

[@bluesquadron593]

By the way, I haven't seen people using Portainer for docker orchestration this much. Pretty much most people are using it for just monitoring. Lately some people are recommending to move away from it altogether. Something about the code owners' direction with features and licensing, I believe. Recently I migrated many of my docker apps to individual LXC containers. I have more fine grain control over them and also if one app goes ape, it does not collapse the whole eco-system. In this case of watchtower is not playing role, but there is the tteck helper script for updates.

[@Lunolux]

i stopped using portainer since they start add "yellow banner" in the top menu, and asking to get a licence was annoying, so replace it with Dockge

[@bluesquadron593]

@@Lunolux yeah, saw this solution too!

[@jasonmhite]

I never really understood why people like Portainer, at least for homelabs. It seems so over engineered, and it makes it annoying to do stuff like making other folders or files outside of the docker compose. Dockge is much more straightforward as a webui for Docker Compose.

[@codeman99-dev]

Watchtower is neat. That said, if you need to keep a small handful of application up to date, don't be afraid to roll your own solution! A shell script that compares the image digest before and after a `pull` is all you really need to get started.

[@wchorski]

I had a strong feeling that a simple shell script check w local vs repo versions was all I needed (compared to all these auto update or monitor tools) could you point me in the direction on how you print out a local container's version and how to grab the latest version number from a remote repo?

[@konitobe-fw8gs]

Updating containers was the reason I avoided docker. Now I have no excuse.

[@suntoryjim]

No thanks. Been burned twice too many times by this. I prefer to be present when my stuff breaks in those instances when latest is not greatest.

[@RaidOwl]

Good point. You can set it to monitor only so that you’ll be notified but it won’t actually do anything.