Best Docker Container Server Setup // Docker Swarm, CephFS, and Portainer

[@RaidOwl]

Awesome video. This is something I've been wanting to try for awhile so I'll be back to reference this :)

[@c0p0n]

I have worked with swarm on the enterprise a lot and it has serious reliability issues. For smaller cluster sizes like 3 nodes at home it might be fine, but at larger deployments of a couple of dozen nodes the problems start to show, and when you approach 50 or 60 nodes it just bursts at the seams. But for homelabbing I'm sure it's fine.

[@ericneo2]

Avoid Yacht for docker, it cannot reliably start and stop containers.

[@VirtualizationHowto]

@c0p0n I am really interested in your feedback on this subject since you have worked with this quite a bit in production. What have you seen to be the failure point? Where do you see issues start to creep in at what layer with the Swarm cluster once you start scaling it up?

[@c0p0n]

@@VirtualizationHowtoa few issues, all related with capacity. Adding extra leader nodes is somewhat unreliable; the networking model is brittle and can get overwhelmed relatively easily and at higher container counts (+200), leaders sometimes fail to fully reap containers, leaving running processes behind until nodes die. But at these smaller scales I don't expect any of that to ever become a problem. It's always been a function of scale as far as I can tell. I've done a few swarm->kubernetes migrations over the past 4 or 5 years

[@fakebizPrez]

Your videos are the best. 👍👍

[@VirtualizationHowto]

Thank you @fakebizPrez !

[@lindermannla]

Excelent video! Congratulations!

[@drbyte2009]

Great video, i was wondering if i can setup a (bridged) firewalls (OPNsense of Pfsense) for high availability with this method

[@PaulLittlefield]

Hi, great video and nice idea with keepalived. I'm looking to improve my cloud hosting and came very close too using Docker Swarm a few months back, but got stuck on the distributed filesystem. So, microceph looks to fix that hurdle. It would have been good to show an actual deployment of an app but this is really useful, many thanks :-)

[@VirtualizationHowto]

@PaulLittlefield awesome! So glad it was helpful!

[@noja7541]

Great tutorial! What's the SSH terminal you're using? Looks cool!

[@VirtualizationHowto]

Remote Desktop Connection Manager and there is a free version as well. Very good product and I use it daily.

[@Felixls]

yeap, I'm running it too for years, but there is one problem is that swarm doesn't work with ipv6 overlay network, so deal breaker for me now as I'm moving away from deprecated protocol (ipv4)

[@Glitch_860]

awesome tutorial. I am running a docker swarm as well. But I did an external VM to host and NFS share for the swarm. It works but I worry about hosting a game server as a swarm service, Don't want to loose my server progress.

[@VirtualizationHowto]

@Glitch_860 ;thanks for the comment. I wouldn't think you should worry if you lose the Docker service. Is the game data stored on a persistent volume on your NFS share?

[@Glitch_860]

@@VirtualizationHowto yes I specify it as a volume in the docker compose file.

[@Glitch_860]

Well in doing come testing. Specifically with factorio. Seems that when I force a reboot of the host running the server. Docker swarm does move the container. But it spins up a new save file. so it is not keeping them persistent. So seems I am missing something.

[@PauloCesarFernandesJunior]

Great video!!!!! Tks a lot! So... after all the configuration of swarm and ceph, i install portainer normally on the first node? that's it?

[@tenekevi]

You can deploy it from any manager node. Once deployed, it runs on the nodes you have configured for it. Start thinking in terms of a cluster.

[@eric-seastrand]

How well does ceph handle file locking? Every time I think I have highly available storage “figured out”, I try a container that needs SQLite and it won’t work for whatever reason. It seems like a lot of containers are built assuming they’ll have direct access to fast storage. That makes it very hard to do HA anything. I’ve tried SMB, NFS v3, v4. Now using GlusterFS and it’s better but some apps still don’t like it. Jellyfin for example. The database wants fast local storage and the app crashes after a few minutes with errors about database.

[@igorshubovych]

Great video! But isn't having an orchestration (Swarm in your case) an overkill for home lab? In most cases, you will be the only consumer of the services you host there.

[@VirtualizationHowto]

@igorshubovych you are right! It is probably overkill for most in their home labs. However, like most things in the home lab, we all do things that are overkill to learn and just have fun. However, on a more serious note, I actually do benefit from what orchestration gives me in my lab. It actually now allows me to dran a node for updates, patches, or other maintenance without disrupting containerized services, like PiHole which runs DNS for my non-production LAN that my family will complain about if down, haha.

[@VoltVandal]

Hi, thanks for this video, one question, is this a possible way to get a kind of "HA" cross datacenters ? (yes, latency, speed might be a problem) but possible? do i need to open all the ports for ceph and swarm ? THX

[@staticalmo]

noob question: how did you find the IPs of the VMs?

[@JamieLloyd]

Hey Brandon, what is the app you are using in this video with the SSH sessions?

[@VirtualizationHowto]

This is Remote Desktop Connection Manager from Devolutions

[@JamieLloyd]

@@VirtualizationHowto Thank you so much for the info, I'm looking forward to to trying it out!

[@JamesGreen-gv4yn]

Curious that you didn't read the line re: "To add a manager node to this swam..." and instead chose to join nodes and then promote them. What do you think is the difference? Even if the results "appear" to be the same.

[@VirtualizationHowto]

@JamesGreen-gv4yn I believe it to be the same. The reason I go this route is to show how to join with workers and then if you want and desire to have more managers this helps to show that progression.

[@matthewmiles6913]

Maybe a silly question, but what If you have your three docker nodes running on ProxMox with ceph. Can you use the ceph pool in ProxMox in any way for Docker Swarm?

[@VirtualizationHowto]

@matthewfiles6913, I believe you sure can with CephFS. I have created CephFS on top of Ceph in Proxmox and presented to Windows, etc...So, for your Docker swarm nodes you would just present the manager IPs to the nodes just like I did in the vid, except it wouldn't be running local, which wouldn't matter.

[@matthewmiles6913]

@@VirtualizationHowto Thank you, I'll give it a try. Great video!!

[@ClayBellBrews]

What about plex and/or other hardware encoding apps that need nvidia?

[@LampJustin]

Honestly i don't really get why you'd want to use swarm, it feels like its on life support and has been deprecated until mirantis bought docker. Almost everyone who wants high availability in their homelab gain so much more knowledge from running k8s that they can apply at their job or job interviews. And everyone else should just use the ha feature of their hypervisor to failover or live-migrate the docker vm in case of failure or maintenance

[@VirtualizationHowto]

@LampJustin I really appreciate your feedback. Honestly, I think Swarm is alive and well. For me, it is about using the right tool for the right job. Is Swarm as good as Kubernetes at some use cases, no. But is Kubernetes always the answer for highly available containers? No, I don't think it is on that front either. I really like that Swarm is a middle ground between a standalone Docker host and full-blown Kubernetes.

[@casey1027]

Is it possible to do the same thing with NFS?

[@LampJustin]

Yes

[@VirtualizationHowto]

Yes NFS is a viable shared storage as well.

[@kgottsman]

@@VirtualizationHowto Careful with NFS... Things that rely on a database (like the *arr stack and Jellyfin) will get randomly corrupted network shares. Requires true block storage.

[@joseph7jk]

What is the name of SSH terminal? It is great!

[@VirtualizationHowto]

@joseph7jk, thank you for the comment, this is Remote Desktop Connection Manager...I love it...maybe a video coming soon on that front, stay tuned.

[@joseph7jk]

Thanks for the info, I'll catch around to see it! ​@@VirtualizationHowto

[@JamesGreen-gv4yn]

Shared storage is a requirement? My Kubernetes clusters do not have any shared storage. So is this just a requirement for Docker Swarm? Can it be configured without using shared storage and if so, what limitations would it impose?

[@LampJustin]

I believe it is not really, but you would need a volume driver for docker that could mount the data otf. Like a csi driver in kubernetes. What's done here is that you do a "hostPath" mount. So that's why it needs the storage to be available on all nodes on the same path

[@VirtualizationHowto]

@JamesGreen-gv4yn shared storage is a requirement if you have/need persistent data (stateful apps), think databases as an example. If one Swarm or Kubernetes node fails, you would want that data to be available to the other nodes to respin that service and still have access to your data. But, you are right. If you are running all stateless apps, there is no need for shared storage.

[@ericneo2]

Lost me at keepalive. The vast majority of people would want to run this in a test/production environment with a gateway and internal static IPs in a load balanced scenario instead of failover.

[@VirtualizationHowto]

@ericneo2 yes that would be true as well. Keepalived is a tool for certain use cases and it is simple. For more robust configurations, you would want to stick a loadbalancer in front.

[@staticalmo]

I expected more angry comments towards Docker Swarm.

[@primedew4501]

First