LXC/LXD vs Docker Which is better?

[@shrimpoffthebarbie]

I switched over totally to LXC. Your explanation ratifies my architectural decision. Well explained.

[@scottibyte]

Be sure to watch my numerous other LXD tutorials.

[@manishkhanna76]

Very useful. Just in midst of setting up a nuc server. This clarifies some of the doubts.

[@scottibyte]

Glad it helped Manish. I have quite a few videos on the channel that discuss LXD and VLANs.

[@Froggie92]

very informative, much appreciated 🙂

[@thenanook]

super clear many thanks

[@scottibyte]

Thanks. Be sure to watch my other videos on LXD & Docker.

[@shunpillay]

Thanks. I found this very useful.

[@scottibyte]

Thanks. Be sure to watch my many other LXD and Incus tutorials as well.

[@elksalmon84]

10:40 This is not a downside. This is the way it's designed for, like Jails in FreeBSD. If i would need a clean OS to work with and maintain, LXC is a smaller and simpler, than a full size virtual machine. In this case Docker design would be a downside for me.

[@computerguy1893]

Great video very informative.

[@scottibyte]

Glad you enjoyed. Join the chat at chat.scottibyte.com/.

[@Ihave3treasures]

Very useful, much appreciated🥰

[@scottibyte]

Thanks. Be sure to subscribe and come by chat.scottibyte.com/ to chat. Lots of LXD content on the channel.

[@Catzzye]

Thanks for information!

[@nadtz]

I'm in the process of rebuilding my little 2 system homelab and have been planning out what I want to do with proxmox. Once I properly understood lxc/lxd vs docker/podman lxc made more sense for most of the apps I want to run that I don't want to run a full VM for, but I can also see value in spinning up a docker container to mess around/test with. lxc/lxd reminds me of jails which I'm used to on FreeBSD, just simpler to deploy via proxmox.

[@scottibyte]

Watch my video about Lxdware LXD Dashboard for a simple and powerful gui.

[@nadtz]

@@scottibyteWill do, thanks.

[@YannMetalhead]

Very good explanation, thank you.

[@scottibyte]

Thx. Hope you watch my other content.

[@kongoulan]

But are multiple LXC more dependent than running one OS with docker in Proxmox? I saw a few people put all applications in multiple dockers in 1 LXC container / VM, while others tend to create multiple LXC container for all their applications on "clean" OS. So which one is better for limited resources and 10 applications. 10 apps in docker or 10 LXC containers. I'm running a remote host, so security is an issue, since some applications are exposed.

[@scottibyte]

I think you are really asking two questions. LXC/LXD containers are not really any more resource intensive than Docker containers. In fact, the original Docker was built on LXC. I know a lot of people that run one Docker host and have all of their containers on it. My philosophy is that if a Docker app has a single container, I tend to run in on my Docker host with other apps. If I have an application that has many Docker containers, I tend to run it inside a LXD container. One example of that is my Bitwarden. My Bitwarden is installed inside a LXD container and that's because Bitwarden has six Docker containers. Some folks would say that if you use MariaDB that you should create a Docker container instance for MariaDB which mounts the database persistent data to some common location. Then, any app that needs a database should have a database instance on this one container. My personal experience has taught me that having each app have its own independent resources works better. One reason that this is the case is that if a typical app uses a LAMP stack and some component of that stack becomes corrupt or relies on a particular version, that app might go offline. If combined, then all apps using the LAMP stack would go offline. So, I like the redundancy. On my main host I have 35 containers. Ten of these are Docker containers at the host level and 25 are LXD containers, some of which may have Docker containers in them. These 35 containers, including a Plex server and a software NVR, use typically 10% of my Ryzen 7, 1700 Pro CPU. That host has 64GB of memory. Most of the containers have a memory cap from 1GB to 8GB and the total memory in use is 30%. So, containers are lean. I believe that the benefits of having a clean slate LXD container for most apps is my go to solution. If I were running on a remote server, I would have that server connected to my network with a VPN. Then, I would have all of my services use reverse proxy in order to reduce my security threat surface to just the proxy server. If I understand you correctly and you are running Proxmox and are thinking of multiple LXD containers and are worried about exposure of these remote instances, I would establish a site to site VPN from where the Proxmox host is located. Put all your containers in a single VLAN and make that VLAN site to site VPN to your main LAN. Bottom line, there is no right answer and I told you what I tend to do and why. Your personal use case might differ. The best part about this type of infrastructure design is that you have almost unlimited options. I focus on LXD on my channel because I believe most folks use VMs and Docker, but either don't know where LXD fits in or they have just never considered it as an option. Thanks for watching!

[@kongoulan]

@@scottibyte Thank you for your very elaborately answer!! I was studying it back and fort in the past days and hopefully can use most of it. When I have more upcoming questions, I will ask again. I think your idea is pretty good to use containers if there are more dependencies than just 1 docker for an app to run. It's good to know that containers don't take extra resources, which i thought before, since many other content creators said, that they are a bit overboard. When I think about it right now, would a database be already a too big of a dependency to consider it in 1 container with a docker. I'm thinking about Nextcloud especially, since it's a pretty simple docker install, but it needs that database.

[@scottibyte]

@@kongoulan Interesting that you should mention Nextcloud. I ran Nextcloud as a Docker for several months and I ultimately installed it as a LXD. My performance is better now and I know it is because of the database installed in the container. I also tend to think, maybe falsely, that applications with more dependencies upgrade more easily in LXD. Perhaps that belief comes from the fact that really with Docker it is all about destroying and recreating the container and just repointing the new container to the persistent data. I've been burned a few times in that regard where an upgraded container is not always compatible with the external persistent data and the whole mess just corrupts. Bottom line is that I love Docker for simple, single container apps without reliance on complex databases. That being said, everyone has different experiences depending on their environment.

[@kongoulan]

@@scottibyte Thanks, that helps me a lot. Now I can continue build up my server because of you!

[@scottibyte]

@@kongoulan Thanks, and that's the reason I do this. For me, IT was the perfect job, because it was and is my hobby. I like paying it forward.

[@joseluis8421]

Thanks, Great video!!!

[@scottibyte]

Thanks Jose.

[@JanDahl]

I'm running a lot of LXCs - including one for my Docker needs 😊

[@scottibyte]

Great! Consider moving on to Incus. Watch my tutorial "Incus Containers Step by Step"

[@JanDahl]

@@scottibyte, having seen the first third, I'm not sure what I'd gain as compared to my current (Proxmox+Portainer) setup

[@scottibyte]

@@JanDahl I guess you missed the advantages. Sounds like Proxmox is best for you. Loading up all docker apps on one VM is not a best practice, but that's just an opinion.

[@JanDahl]

@@scottibyte For the past week I've sporadically tried to look up what differences I'd experience but I'm still not sure what you mean. Are you suggesting running Incus alongside Proxmox and have Proxmox handle the VMs and Incus handle the LXCs? I also do not understand how that would help with regard to "best practices" as I'd be just as "vulnerable" to server down no matter which tech I'd put my Docker containers in. I am using Portainer to "orchestrate" but I've noticed that it has a weird Achilles heal in that the "orchestrater" doesn't keep a copy of the compose config and if the link to an agent has been severed and has to be reinstated, it lacks a lot of config control - but I can't see how Incus would help that. I'm not trying to be flippant :) I could only find a German video about "Proxmox vs Incus" and I admit that my German is quite rusty indeed.

[@scottibyte]

@@JanDahl Be sure to join the chat at chat.scottibyte.com/ because asking questions here is awkward. Proxmox supports lxc containers and VMs. Incus/LXD servers both have linux containers (lxc) as their underlying infrastructure. Incus/LXD servers add an API and a CLI to lxc containers. Incus/LXD servers both support virtual machines. I prefer running Incus/LXD over Proxmox mostly because Proxmox does what it does in a sort of "proprietary" way despite being open source. It's just my opinion, but I see Proxmox has more of a blackbox to manage VM's and lxc containers, whereas I see Incus as a truly open source solution that exposes the actual commands and allows the end use more control over the functions. I also believe that nesting a single docker app in an incus container provides a better degree of isolation and security in regards to the stability of docker and I have several videos on that. In short, join the chat and come by and ask questions.

[@Being_Joe]

Great explanation. So if I understand this LXC/LXD is similar to FreeBSD Jails or Solaris Zones.

[@scottibyte]

I would say, a little bit. LXC/LXD is a containerization technology making use of prebuilt images for the rootfs and shareing the parent OS kernel. So, its far more sophisticated than "chroot" type technologies. The newer LXD which I focus on in quite a few of my videos is very powerful due to its CLI/API that allows granular control over network, storage and other devices presented to containers.

[@ewenchan1239]

Great video!! I am in the process of migrating all of my Linux VMs over to LXC containers, because they're lighter weight, which leads to a more efficient use of my physical/hardware resources. For my homelab use, I am not worried about any potential snooping or such secuity concerns, but I understand that for Cloud providers and/or corporate/enterprise users, they might NOT want to necessarily use LXC/LXD containers.

[@scottibyte]

LXD provides all the security required to secure projects/containers. It's up to the end user to use it or not. I have over 150 videos on the various aspects of LXD and the newer open source Incus container project. Watch both my LXD Containers Step by Step and my Incus Containers Step by Step.

[@ewenchan1239]

@@scottibyte Thank you. I'll have to look into that. I did run a quick search on your channel for "LXD Proxmox" and I'll have to watch it with the sound on later. I'll have to look. I think that with LXC sharing the kernel with the host -- there will be a risk for "snooping" between LXC containers. For homelabs and home users -- probably not a huge deal (unless they get hacked).

[@marcello4258]

Wow this was a good overview! Thanks. I am using LXC and jails for some time. Never really used docker as I did not feel the need yet. One thing was not clear for me after all. Is the docker runtime coming with it's own kernel or does EVERY container ships with it's own kernel? - the latter would have a really big footprint imo. I was wondering since you pointed out the kernel issue with lxcs which in fact is a fair point, but on the other side I was assuming same goes for docker that it shares the hosts kernel.

[@scottibyte]

Docker and LXC/LXD both share the kernel of the host machine. Virtual Machines each run their own software kernel though. Containers are lightweight because they do share the host kernel. That's one of their superpowers, although that can be a weakness if the host kernel is missing a key feature that you want a container to have.

[@marcello4258]

@@scottibyte yes, that is right! But my question was how docker handles the kernel business? You said the missing kernel functions are effected only w/ LXC not w/ docker itself :)

[@scottibyte]

@@marcello4258 So, if a Docker container calls a kernel based function that is not present in the host kernel, it fails. For example, a Docker container on an older host kernel might not be able to use cgroups.

[@marcello4258]

@@scottibyte ok then this part is quite confusing in the video, to me it seemed this issue is only present within lxc/ds

[@knofi7052]

Anyway, I prefer using LXD containers because Docker containers are not persistent and it gives me the feeling of higher flexibility when using it more like a virtual machine. However, Docker containers have some advantages in specific situations.

[@scottibyte]

Quite true. Docker volume persistence is available through volumes. LXD are a lot like virtual machines, but without the bloat.. Watch my many other LXD videos where I go in depth on networking and vlans.

[@dtesta]

What exactly is preventing you from making a docker volume for the entire root and simply pick a whole OS as the image? Would result in pretty much the same thing as LXC...

[@llortaton2834]

What about docker in LXC? :O

[@scottibyte]

Watch this one. kzbin.info/www/bejne/jaWlm4KvrcmebLs

[@jsniderhan]

Very good explanation sir. I was wondering the difference, and i even learned some stuff about docker that i didnt already know.

[@scottibyte]

Thx for the kind comment. I used to do more Docker than LXC/LXD. I have now focused on LXD because of the awesome power with its CLI. I probably had nearly 30 LXD videos now for that reason.

[@jsniderhan]

@@scottibyte I'll check them out. I was debating running all my apps for Usenet in a Ubuntu server w/docker vm on proxmox, but now may consider a few different lxc for their ability to update and backup..

[@scottibyte]

@@jsniderhan Come over and chat on chat.scottibyte.com/

[@billc3114]

Docker sounds like it quickly becomes not worth its' effort in learning. Unless you are a big Corp or development group that shares code or OS/Lang aspects among large groups. But is a very impressive premise.

[@scottibyte]

Bill, a lot of people prefer Docker because deploying apps is so easy and there is a ready library available. I use docker occasionally, but I choose to nest it inside of LXD as I have shown in some of my other videos. I absolutely am hooked on LXD containers because they are simple to deploy and very lightweight. It's like having a VM without the performance and resource cost of a VM.

[@Alphahydro]

Never really knew what all the docker craze was all about. For the most part you have to run the docker instance inside a container anyway.

[@scottibyte]

Docker is a Container. LXD is a container. Docker is a container that virtualizes an application. LXD is a container that virtualizes the OS. Docker is not run in a container because it is a container.

[@ronaldm.1556]

@@scottibyte yes, docker can run inside a container.

[@scottibyte]

@@ronaldm.1556 Docker is a container and it can also run inside a container. For example, I have apps where the app is run inside of an LXD container, but the components of the app are several Docker containers. The key takeaway is that LXD are containers that virtualize the OS and Docker are containers that virtualize only the app. I prefer LXD containers over Docker when possible because Docker containers are not persistent. That means that all non-volatile data must be mounted outside a Docker container and a new version of a Docker container necessitates destroying and recreating the container. Despite having my data mounted externally for these Docker apps, I have lost data more than once when upgrading a Docker container. That being said, Docker containers are VERY easy to use and work efficiently for static applications.

[@ronaldm.1556]

@@scottibyte docker and all modern containers are designed to be stateless, you can containerize your app and parametrize your applications to run on scalable infrastructures, in this way of approach, it enables you to store data of these containers on scalable storages

[@ronaldm.1556]

@@scottibyte comparing docker and lxd is wrong because they are not made for the same purpose

[@GadgeteerZA]

I think the downsides to Docker may also be mitigated somewhat in that: 1. Doing update from within Portainer GUI is just click on recreate, toggle switch for download fresh image, and OK. So can be pretty quick and seamless. 2. The external mounts of a volume are down once only on the container creation, so don't get in the way later. 3. Very true on orphans which should be cleaned once a month or so. 4. The database dependency could be a "feature" too, as I created one container with a database with phpMySql, and now any new containers I prefer to point to that "external" database container because it means I can do all db maintenance on a single db instance, instead of across all containers. 5. Some docker containers can run app updates eg. my Wordpress one, so I do not pull new images for that container and ignore checking for updates for it. But yes not OS upgrades. But thanks very interesting video and I'm going to look a bit further into LXC/LXD now. True, it's pick the right tool.

[@scottibyte]

I have a video on LXD Dashboard (kzbin.info/www/bejne/Y33KlHeZd56CnNU) which is a GUI for LXD that does what Portainer does for Docker and more. My personal experience with Portainer has been lackluster because it tends to have sketchy stability for me. The docker containers themselves run fine. My attraction to LXD is really all about having super lightweight OS instances that I can snapshot and export and even upgrade. In fact, I tend to nest my docker apps inside LXD instances as I have shown on my latest videos. Thanks for watching BTW!

[@knofi7052]

You can use a single db instance with LXC/LXD as well if you want.😊

[@HatingDrink]

7319 Heaney Ports

[@scottibyte]

What are Heany Ports?

[@paulwratt]

You forgot to mention the spin-up speed of each (especially in a CI type situation), response time can be a major factor depending on your use case and the prerequisits for your application

[@scottibyte]

There is virtually no "spin" up time difference between LXD and Docker. Neither go through a boot cycle unlike VMs. The complexity of the application makes all the difference in start time. In another of my videos I address VMs vs. containers. I have some Docker containers that take much longer to start than some of my LXD containers and vice versa. I didn't get into CI/CD issues because although that is a concept to software deliver management (SDM), it is not critical for SOHO users to understand. Glad to see that you understand the technology and the considerations.

[@paulwratt]

@@scottibyte ""There is virtually no "spin" up time difference between LXD and Docker. "" - not according to the creator of `pistion`, YT v=SD4KgwdjmdI time 3:00, where the discussion he goes into may also reflect certain CI/CD environments - Yes, I still think there are some considerations to the application of the container usage. As for SOHO users, I am sure that if they are using Docker or LXC/LXD then they will be(come) aware that certain of their application _may_ require repository builds (depending on the application and the users use case), in which case CI/CD is worth investigating. I could not find any further details on the `piston` public api server, but it appears _hardening_ was also improved via LXC/LXD (based on the content of the video and an associated blog post)

[@dtesta]

@@paulwrattContainer are just glorified "chroots". Pretty much all they do is to download a "root filesystem", unpacks it at some path and chroots into it with some cgroup rules. There is no traditional "booting" happening.