Containers Are Not VMs! Which Base Container (Docker) Images Should We Use?

[@DevOpsToolkit]

What do you use as base container images?

[@zygote846]

Alpine 😢😢😢😢😢

[@hkupty]

I'm using default openjdk base images. Might play around w/ jdeps+jlink and scratch..

[@DevOpsToolkit]

@@zygote846 Alpine is great!

[@MatteoVeroni]

I use multistage builds. I work using Java. At the moment I create a minimal app with jpackage (minimal jre and executable), then I add the deb installer to a Debian minimal image and install it with dpkg -i. Finally I copy all to a scratch image stripping away all the unnecessary things from debían. If there are better approaches I'm open to suggestions

[@DevOpsToolkit]

@@hkupty I would recommend Chainguard images instead. They are compiled daily and have a guarantee of zero vulnerabilities.

[@hkupty]

I surely would love to learn more about ephemeral containers :)

[@DevOpsToolkit]

That's all I need. I'll work on it and try to publish it next week.

[@RafaelDurelli]

You are the best 🎉🎉🎉🎉

[@momusi.9819]

@@DevOpsToolkit Thank you!

[@cwmoriarty]

@@DevOpsToolkitLink it here?

[@DevOpsToolkit]

Here it goes: kzbin.info/www/bejne/p3zFZ5-ldqqDjJI

[@teddybear7949]

A demo on scratch image + ephemeral container to debug pod from scratch would be great. As tou proposed it 😊

[@cheebadigga4092]

+1

[@caseyphillips]

+1 Yeah, a simple run through of how to do this in a graceful manner would be lovely!

[@DevOpsToolkit]

Coming next Monday :)

[@cazador517]

I agree that many image come more bloated than they should, BUT scratch should be used with care. Scrtach images literally have nothing in them, but a lot of apps expects some things to exicts in certain places, and if they don't things tend to get wacky. Namely when you create a image from scrath you need to make sure that you have a valid /etc/passwd, a /tmp directory, a valid /etc/localtime and a ca-certificates (this one is only needed if you app make TLS connections, but most will). That's the bare minimum, on top of that you need to put all your app¡s dependecies. And you have to keep in mind that whatever your throw to your image, you have the sole resposability to keep it up-to-date, which can be no fun at all, and a lot of work. IMO the best you can do is just forget about all that and use Google's distroless. They have many flavours, do you have a statically linked app that have no depoendcies, not even libc? Use the static variant and you get basically what's an scratch image but with the must have that I commented earlier. What!? That you do need libc, and also libssl and openssl like 99.9% of the programs out there? Do not fret, just use the distrolless/base (or even /cc for libgcc support) mage and you are good to go (BTW, most of the time you should still use the "base" image for go apps, beacuase Go can use some libc functionality if present, giving slighly better performance and a behaviour more inline wiht the rest of apps). That you rather use Node or Java? Why of couse, just use the /nodexx or /javaxx images and you have everything you need. Python you say? Hmm, that's where it gets tricky, there is an /python3 image, you see. But I'm afraid that that's only experimental and using it on production is ill-advised. Ah, I must not forget to tell you that these images use glibc so no musl edgecases for you and if supply-chain security it's a must for you, you can easly check these images with cosing.

[@DevOpsToolkit]

True. I tend to fall back to Alpine or Wolfi when things with scratch get complicated.

[@AdrianMouat]

This is exactly why we created Wolfi & Chainguard Images. If you use our static image, it's basically the minimum you need for the majority of applications. By default our images don't have a shell or package manager, but there are -dev variants with those tools in case you need to add things etc.

[@mirceanton]

I'd definitely want to see a video on ephemeral containers!

[@zygote846]

Great content as always Viktor.I would really to get more insights into emphermal containers :)

[@DevOpsToolkit]

Noted! It's coming next week.

[@ericsmalling]

Just stumbled across this, as always, great stuff Viktor! One nit: Wolfi images come in two variants, latest and latest-dev. The non "-dev" ones usually have no shell, package manager, etc. They are similar to distroless. ("-dev" tags have those tools). Dislaimer: I work for Chainguard

[@fenarRH]

Universal Base Image (UBI) is a good choice if you like to go explore (free) mode then switch to enterprise mode with sla-backed support easily.

[@ferozekhan267oa]

Great insight for anyone who has drifted from the best practices. But most container based training videos don't focus on best practices either, as their objective to only successfully run their use cases limited to the video tutorials only. Having said that, how about a demo video on using scratch images for running binary application + ephemeral containers.

[@sohansingh2022]

I just cant appreciate you enough!

[@Modzybear]

Debian, Ubuntu, and other Linux Docker images are perfect for their intended use case: Developer Containers. Specifically because you are attempting to replicate a local development environment across machines with zero friction; native support in VSCode, no learning curve for another technology like Vagrant, and identical development environment in Cloud-based code editors like Codespaces and Cloud9.

[@Justjames283]

Yes! Let's see more about ephemeral containers

[@DevOpsToolkit]

Coming next week.

[@mondo1926]

A video on ephemeral containers would be great to see. Also, configuring k8s probes with scratch images in painful and we can't use alpine images in production.

[@laughingvampire7555]

well, in Linux, Docker containers are not vms, in MacOS and Windows they are installed in a VM. sadly the industry has forgotten about chroot and vservers. and nixos does it better than docker, because all you need is the configuration file and your app, nothing more. Nixos can also use docker if you want to and it can create alpine images even smaller than with docker. I share this because well, is just amazingly productive and issue solver for my experience in the last 5 years of using docker in established teams.

[@pististech-hub]

Big Thank you @DevopsToolkit. I would love to see a demo on ephemeral containers. ❤

[@OldG4merDad]

OMG, my head exploded. Where is your class to relearn all this stuff??? Good video, I would love to hear more!!!!

[@VladimirVyazmin]

Yes, please create video about ephemeral containers.

[@IvanRizzante]

Thanks for another interesting video on an interesting subject 🎉 I totally agree on using multistage build images and avoid the unnecessary stuff in containers. For java we use azul jdk alpine based images even though tools like jlink exist to build a custom runtime image based on java modules for scratch. It's also worth mentioning distroless images which allows node and Java applications to run without extra dependencies, just google for them. Final note, I think that a video on ephemeral containers is a very good idea

[@n0kodoko143]

Ephemeral container attachment, plz!! Awesome video btw!

[@brandonwood6900]

Definitely want to here more about ephemeral containers

[@DDD-d1g]

Great video! As you mentioned, "from scratch" is good for binaries like Go, Rust, C, C++, etc. But what about dynamic languages like Java, DotNet, Python, JavaScript, and so on? Is it related to Google's "distroless" base images? And how about compiling Java apps using GraalVM to binaries? (maybe there are similar tools for other languages as well.)

[@DevOpsToolkit]

Compiling is not a problem with multi stage builds. I'm ok using any base image in all but the last stage that forms the final image. As for dynamic languages... I suggest Wolfi and other ChainGuard images.

[@chasim1982]

Hi! Great content as always, I am using alpine as base image & using slim tool kit to reduce the size & image hardening, I would appreciate your opinion on slim 🔧 tool kit, can you please make 📹 video on slim toolkit, thanks ❤

[@DevOpsToolkit]

I haven't used it recently so I'm not sure how it works now. Back in the day when i dived into it, it worked only with docker which was a deal breaker for me since i could not run it inside my clusters.

[@chasim1982]

@@DevOpsToolkit Thanks for the reply 🙏 👍❤️ you are right still the same, we can not run inside our cluster, that portion of making our images slim needs to be done outside, I tried it & it worked, kindly have a look & suggest #(kzbin.info/www/bejne/fXWVlpmIeJtof68)#

[@CallousCoder]

I use alpine small and light. And works great with Java Spring boot, rust bins, c/c++ bins, and even the horrible interpret scrap like NodeJS and Python.

[@KingoOoVideos]

Thanks it's a great topic. I think with source to image (S2I) & Buildh the process of creating image from scratch is simplified

[@abessesmahi4888]

Please a demo on scratch image but please use an interpreted language instead of compiled on ( because the compiled one are easy to deal with) Than a demo on using ephemeral container ( I would like to know if the ease a benefit of using them for me who uses docker-compose only for deployment). Thank you so much for your efforts.

[@aviadhaham]

Yes, please make a video about Ephemeral Containers!

[@KK-4ever]

I think alpine in rootless and read-only mode containers are a good config to start with for nodejs/python based apps.

[@bigfishoutofwater3135]

The concept of it is more secure if you only include what you need reminded me of when you talked about Talos. Seems like using that for K8s would pair nicely with scratch-based containers.

[@DevOpsToolkit]

Yeah. The same logic applies both to VM OS and container image.

[@Tszyu01]

This was a great video. Yes to a ephemeral containers video.

[@dougsellner9353]

What tool/gui shows your registry/security scan/vulnerabilities? (THANKS!)

[@DevOpsToolkit]

I tend to enable security scanning in container image registries. In my case, i use container-registry.com, but others should have it as well.

[@edenr1988]

Great video, well explained. What about those container images offered by Google - distroless ? I'm having difficulty understanding why to use them over scratch ? Is it just because of them being more debugable ? I know also there is a command kubectl debug which allows me to spin up a container like you mentioned inside the pods so I can troubleshoot networking issues. So I'm still confused why people still using those distroless container images from Google over scratch for compiled applications. Any other particular reasons you can think of ? Thanks!

[@DevOpsToolkit]

Sone use "other" images because they are used to them from when they run apps without containers. Others use Alpine, Google images, or Wolfi because they cannot use scratch (it does not work well for many use cases). Finally, there is, as you mentioned, the need to debug containers that requires certain tools to be available. When scratch is not an option, i recommend Wolfi and other ChainGuard images. Alpine has a few issues. Google is an ok option but Wolfi is still a better choice. As for debugging... Wait until the next week's video 🙂

[@123unhooked]

ephemeral containers is a yes for me. would love a video!

[@JimVincitore]

Great video. Yeah but...what if the underlying Linux OS is updated and breaks your App in your Docker container? I thought the whole idea of Docker was to prevent this.

[@DevOpsToolkit]

Would packaging OS in a container image prevent that? Containers depend on host OS no matter what we put inside container images.

[@brazenintellect364]

I would greatly appreciate you covering Kubernetes device operators e.g. nvidia container toolkit or intel device plugin etc and how time slicing can be implemented using them.

[@DevOpsToolkit]

Adding it to my to-do list... 🙂

[@santoshperumal129]

Excellent video thanks for doing such videos please keep contributing we wait for your such videos

[@Wector11211]

What if some official approved containers use different distros as a base image? For instance, mysql uses debian, while mongodb uses ubuntu and I need both services in my network, so I will have extra overhead from both distros. Does it mean I should create my own containers for both services (from alpine) in sake of minimization?

[@DevOpsToolkit]

I would still use the official images rather than go through the hustle of creating and maintaining my own. As a side note... Those two are "special" for not offering at least alpine versions of their images.

[@blaiseutube]

Me before starting the video: "The best base image is "scratch" Victor in the first 5 minutes: "The best base image is scratch "

[@gorhal19686]

Great video as always :-) What container registry are you using?

[@DevOpsToolkit]

I'm using c8n.io from container-registry. It is managed Hsrbor.

[@MrKamikadzer]

Alpine has pretty nasty downside. It's use musl instead of glibc which gives problems in very uncommon places

[@DevOpsToolkit]

I believe (but I'm not 100% sure) that Chainguard images use glibc.

[@AdrianMouat]

@@DevOpsToolkit You can be 100% sure :) We compile everything from source against glibc

[@michaelraasch5496]

That is very insightful. Does anyone have any experience with using nvidia CUDA drivers with scratch or alpine?

[@DevOpsToolkit]

I'm not using CUDA myself so I might be wrong but, from what I gathered, it is compiled with glibc. If that's the case, it probably won't work with Alpine but it should work with Wolfi (at least the glibc part).

[@mrgdevops]

YESSS for ephemeral video :)

[@squalazzo]

yes, go for ephemeral containers!

[@SumanChakraborty0]

Would love to go with any distroless image !

[@cheebadigga4092]

Thanks for wolfi! Didn't know that one yet.

[@DevOpsToolkit]

ChainGuard is doing a great job with images.

[@stanrock8015]

Big fan of distroless but not always the best fit.

[@roguechambers]

yes to ephemeral containers as a next video

[@agostinoleoni6696]

With Java it's not so simple, and I think that alpine of similar (I use ubi on openshift) are ok. But I think that the ephemeral container point could be moreinteresting, Have you any plan to make a video about?

[@DevOpsToolkit]

I do. Video is coming this Monday.

[@mehdimosbah3221]

Hey Victor what about "Distroless" Container Images for interpreted languages?

[@DevOpsToolkit]

They are definitely a better choice than Ubuntu and similar images.

[@DynamoPhreak]

Thoughts on Rocky Linux? really liking it lately~

[@DevOpsToolkit]

I haven't used it (yet).

[@maxreuv]

A jewel video! Instant keeper

[@popovicmiro]

Hi Viktor, yes plz, video on ephemeral contaners would be usefull.. tnx

[@lamnot.]

How do you avoid similar bloat in a VM, if running it using kubevirt or directly on hypervisor?

[@DevOpsToolkit]

There are optimized Linux distributions you can use. It all depends what it's for.

[@mvkrishna86]

yes, I want to know about ephemeral containers.

[@EdCardinal-MindThump]

Ephemeral contains are fantastic! Please make a post !!

[@maousama941]

finally someone spoke my mind

[@Blindstick1]

WASM Please take us to the next level already 🤘🤘🤘

[@DevOpsToolkit]

Wasm has the potential to improve the situation but needs to build an ecosystem to become a viable option.

[@duckydude20]

if i understand correctly with scratch we cannot have dynamically linked executable? also, how does layering works here? i don't know much about the layering concept tbh. spring recommend putting dependencies in different layer. if you have done any video/blog on that can you link that also?

[@DevOpsToolkit]

Dynamically linked executables are painful and, if i do need them, i tend to switch to Alpine (in the past) abd Wolfie (currently).

[@MatteoVeroni]

Yes for ephemerals containers . And how to debug outside k8 (plain docker)?

[@DevOpsToolkit]

I will publish the Kubernetes part next week and add plain Docker to my TODO list.

[@MatteoVeroni]

Thanks

[@ryanseipp6944]

Curious how you handle ca-certificates in scratch? Maybe its enough to install it from Alpine and copy it to the final image, or should they be mounted via volume in case they update?

[@DevOpsToolkit]

Internal certs with a service mesh while external with cert manager.

[@ryanseipp6944]

@DevOpsToolkit That makes sense. I was thinking more about a service that has an Http client to an external API, but needs cert roots to validate SSL. I suppose the service mesh/sidecars would handle that, though wondering for the case when you don't necessarily want the overhead.

[@DevOpsToolkit]

If you do need "special" that you cannot get from outside the container (e.g. sidecars), scratch is not a good option. In those cases, i tend to fall back to Wolfi (previously Alpine).

[@RumenNikiforov]

I have that use case and i've solved it by installing ca-certificates on the builder image (as part of multi-stage build) Then copy the ca-certificates bundle onto the scratch final image COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

[@yash1152]

what do you say about devcontainer-images that is, VS Code development container image? these are used in github codespaces for doing development, not just _CI_ or _testing_ the default devcontainer supplied by github is based on ubuntu, which uses out of 32 GB: 46% (~14GB): huge.

[@DevOpsToolkit]

Development is different and I'm much less "radical" with it. It's ok to package everything you need to develop, at least from th security perspective. Also, they are vendors so they need to have things everyone needs. It would be better to build your own images though.

[@yash1152]

@@DevOpsToolkit yes, my question was about that itself. that is, what will you suggest as a good lightweight base for devcontainers? > _"It would be better to build your own images though."_

[@DevOpsToolkit]

@@yash1152 for those cases i tend to use whatever is available by default. My focus is mostly on production.

[@yash1152]

@@DevOpsToolkit ohkayy. thanks 👍

[@davemeech]

Slow Jack Nicholson nod for the video on ephemeral containers!

[@urzaaaaa]

Can ephemeral containers be used without kubernetes with just docker?

[@DevOpsToolkit]

Ephemeral containers are specific to Kubernetes (excluding other solutions with the same name). I'm not sure that something similar would even be possible since Docker does not have the concept of Pods (multiple containers grouped together). On the other hand, I'm sure that there are ways to debug Docker containers without "bloating" them with unnecessary tools. However, I don't use Docker in production so I can't say what those tools are (nor whether they exist). When working locally (not production), I never had the need to go for optimized slim images.

[@ogi011]

I can agree with this but i must admit in large complex environment it is very difficult to do any real troubleshooting without having tools built into the image, network tools such as ping telnet ssh traceroute tcpdump and so on, i guess this is not a best practice but i find it easier when you have proper tools within images it speeds up troubleshooting.

[@DevOpsToolkit]

That will be the subject of next week's video :)

[@nickeldan]

In my case, my C binary uses a function available in a particular version of glibc. This is why I use Ubuntu 22.04 as even earlier versions of Ubuntu don't have a new enough glibc.

[@DevOpsToolkit]

Is that version of glibc not available through Alpine or Wolfi? If not, than it makes perfect sense to use something like Ubuntu. The message in that video is not to never use anything but Scratch, Alpine, Wolfi, and similar base images but, rather, that others should be used only when there is no other option.

[@nickeldan]

Not that I can tell. The function I need, sigabbrev_np, was added in glibc 2.32.

[@DevOpsToolkit]

@nickeldan Wolfi images tend to have zero vulnerabilities and that often means that they are on the latest version of whatever is in them. You might want to check it out.

[@nickeldan]

I'll take a look. Thanks!

[@nickeldan]

I was able to get my Docker image from about 80 MB to 14! Thanks!

[@sebiuo5817]

updating my images 😅 grande!!

[@burtberkful]

What about distroless images by Google, how do they compare to Scratch, Alpine and Wolfi?

[@DevOpsToolkit]

I haven't used those so i cannot compare them 🙁

[@Babbili]

one of the best videos

[@itsdenga]

Real question is : How would I convince a project manager that existing images needs rework, especially when it is also running startup scripts which should be part of the build process 😅 This is the real tricky situation.

[@DevOpsToolkit]

People tend to be scared of security issues. Show that managed that there are vulnerabilities and than ask whether he/she would like to have it without them.

[@ioannisgko]

Excellent video 👏

[@abelsalgadoromero4402]

I understand the messages of knwow what you include, don't add extra stuff and most images are bloated. But there's some miss-information included that I doubt you are not aware of. I want to believe the push for views forces ppl to make controversial content. First, even binaries depend on OS libraries like glibc/musl. Go likes to say they generate fully static binaries, but it's not true, it's easy to check with `ldd`. Not addind those into the image means you'll be using the ones provided at the host which can make your app crash if there's any incompatibility (see glibc vs musl). That destroys the purpouse of generating images that are 100% assured to run. Then, that also means you are bound to host updates for CVEs. You won't see them in your scans but just because you are using the host libraries, so it's more like you don't see them.

[@DevOpsToolkit]

That was my bad for not explaining it well. The message should be, "have only what you need and nothing else". Now, whether that is scratch, or Alpine, or Wolfi, or something else depends on what you need.

[@ericegan5707]

Yes on ephemeral containers

[@georgeanastasiou2680]

The video title is sort of misleading, indeed taking off the layers that are not needed using multi-stage builds to process building or even using a stage to pull e.g an artifact, extract it, and only keep in later stage the extracted part you need it does not mean you will not use a base image you are comfortable with through the build stages. :)

[@DevOpsToolkit]

Yeah. I should have explained better that i was referring to base images of final images. My bad...

[@georgeanastasiou2680]

@@DevOpsToolkit The video though is quite good and helpful, thank you for that!

[@hooverzavala4560]

12:21 yes please!

[@DevOpsToolkit]

Already done and released yesterday. kzbin.info/www/bejne/p3zFZ5-ldqqDjJI

[@valtersilva5386]

Hey mate, I enjoy your videos! May I suggest doing one about VSCode DevContainers? I've been using them for the past year and they are very useful!

[@DevOpsToolkit]

Great suggestion. Adding it to my to-do list... 🙂

[@datogokadze]

what distros should we use for VMs?

[@DevOpsToolkit]

That depends on what you're running on those VMs. Talos is a good example of container optimized OS.

[@datogokadze]

@@DevOpsToolkit I run fedora coros for containers didn't knew about Talos thanks will check

[@ParimalaRao-pk2ub]

What about RHEL UBI images?

[@DevOpsToolkit]

I prefer chainguard images.

[@MrSchmichri]

YES ephemeral containers!!11 now ;-)

[@noureddineennacir515]

Can you make a video on Windows containers? You have to use a base image :(

[@DevOpsToolkit]

Unfortunately, I don't use Windows so I'm not experienced with it enough to pick it as a subject 😔

[@lerneninverschiedenenforme7513]

yes, ephemeral containers. and yes, why are os images bad for VMs?

[@DeltachaosTV]

What you have not mentioned is that for running applications written in python or php you should use the php or python or whatever other language yoe are using base images instead of trying to install the interpreter into a alpine or debian based image

[@DevOpsToolkit]

True. I use ChainGuard images for those cases.

[@NicolasFrankel]

For once, I disagree with you: I've had issues with Alpine when running JVM. Either the container took very long to start (say 10 to 20 seconds) or worse, it was stuck and never started. I'll never trust Alpine again (and that's not even mentioning the glibc vs. musl gap)

[@DevOpsToolkit]

Have you tried ChainGuard images? They are my "Alpine replacement" images.

[@anthonyrussano]

yes please!!!

[@AlexApetrei]

Yes

[@venkatamutyala8295]

Ubuntu4lyfe.

[@elainemattos-qo1tr]

yes

[@rhb.digital]

already using Alpine sooo

[@cyberbrainmfk]

Next breaking news: "Debian, Ubuntu and CentOS communities rise protests against Viktor Farcic" :DDD

[@APiratesRevenge]

Has Distroless fallen out of favour?

[@DevOpsToolkit]

The only truly distroless image is `scratch`. Nevertheless, people tend to call "small distros" distroless and they are a great choice when `scratch` is not an option.

[@carlossanchez-ij5sf]

please, go ahead with ephemeral containers 🙏

[@OscarCassetti]

+1 on emphemeral containers

[@TomJordan741]

A video on Ephemeral Containers would be great.

[@pierreancelot8864]

Containers are not VMs....... Then comes LXD/LXC :P

[@julianomoraisbarbosa]

# til

[@crackerjackmack]

You missed that glibc v.s musl as a reason to not-use a small or scratch image. Generally, not needed, but I've run into poorly behaving memory allocations resulting in truncated data.

[@DevOpsToolkit]

I did mention Wolfi whenever I mentioned Alpine, which does solve that problem. Still, you're right. I should have explained it better.

[@crackerjackmack]

@@DevOpsToolkit Ah, I didn't realized that was the biggered difference with wolfi! Thank you for pointing that out.

[@DevOpsToolkit]

@crackerjackmack there are few other differences. ChainGuard images are rebuilt daily and tend to have zero vulnerabilities.

[@fagobg]

+1 for ephemeral containers

[@joebowbeer]

Yes

[@sebastianmarynicz7367]

yes

[@memo.i.garcia]

yes

[@GernotPokorny]

yes