Kaniko - Building Container Images In Kubernetes Without Docker

How do we build container images inside containers running in a Kubernetes cluster? Docker is a bad option since it cannot run inside containers. Using it requires communication to the Docker engine running in cluster nodes. Even if that security concern is not enough, Dockershim is deprecated in Kubernetes 1.20 and will be (or already is) removed from 1.22. That means that alternative container engines like ContainerD will take its place. In other words, using Docker is not even an option anymore.
Kaniko comes to the rescue. It is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.
Timecodes ⏱:
00:00 Intro
04:55 Cloning the demo app repo
07:38 Building container images with Docker locally
08:15 Building container images with Docker from a container
12:11 Building container images through Docker socket
16:16 Building container images with Kaniko from a container
➡ Gist with the commands: gist.github.com/627fcfbfbc17a...
🔗 Kaniko: github.com/GoogleContainerToo...
📚 DevOps Catalog, Patterns, And Blueprints: www.devopstoolkitseries.com/p...
📚 Books and courses: www.devopstoolkitseries.com
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
➡ Follow me on Twitter: / vfarcic
➡ Follow me on LinkedIn: / viktorfarcic

[@manidevops8501]

I was very much excited on seeing this video. i do run all my CI/CD pipelines with docker sock bindings. When the announcement came from k8s community. i was searching for an alternative way to do this, keeping cost optimization also in mind. Thanks for wonderful video. Cheers!!!

[@itsjakepage]

Great video Viktor, Docker deprecation as a runtime on Kubernetes is not the end of the world and this explanation on a possible alternative is super helpful. Cheers!

[@AliAbbas-vp4bm]

This is by far one of the best explanations of why/how to use kaniko within k8s a cluster to build docker images

[@EduardSchildner]

Great walkthrough, well done! I know kaniko for some time but I wish I came across a video like this one a year ago. Well, at least I will start pointing other people in this direction because kaniko can barely be explained better to beginners👍

[@DooMonTouR]

This is everything i needed today, thanks so much This has the explanation of what's wrong, good demo, bad practices explained and it's so easy to understand. Very good job thank you so much

[@tapindersingh2616]

I recently came across your channel, and I am already a fan of the way you are explaining. Thank you for sharing the knowledge.

[@DevOpsToolkit]

You're very welcome!

[@trickytibo]

Great video. I heard of Kaniko but never get the chance to try it. Thanks to you, I will test it right away for building my image !

[@JaydeepDave12]

First ever DevOps channel I subscribed to!! I generally don't like any other channel. You are the best :)

[@DevOpsToolkit]

Welcome aboard!

[@hannahgibson3756]

Best explanation of kaniko I've seen so far.

[@DevOpsToolkit]

Wow, thanks!

[@AnadoxX]

I need to learn much more about Gitosps in my new Job, and I just wanna thank you for helping me out a ton :D

[@vladf7180]

Thanks for the video, it is very helpful for me for understanding what kaniko is used for.

[@BrennoLeal]

Thank you for this awesome explanation!

[@MrRfinner]

Viktor, Oh Yea! You nailed it.

[@sep69]

Thank you for this great video ! Exactly what I was looking for :) I've been watching some of your other videos also and you have a very good way of explaining thing. Thanks again and keep up the good work :thumbsup:

[@DevOpsToolkit]

Glad it was helpful!

[@juanchavaco2378]

Thank you, great video!

[@RideLikeAChamp]

Superb, the secret GCP used under the hood of cloud build. It also facilitate/liberate developers from building docker file. RIP docker in Kubernetes

[@bled_2033]

Crystal clear explanation!

[@DevOpsToolkit]

Thanks

[@rajendragosavi2233]

Excellent Stuff! Superb. Thanks :)

[@DevOpsToolkit]

Glad you liked it!

[@systementorlearning8685]

My experience is that moving build pipeline into the cluster is not that prioritized...at least not yet - however guess that depends on the companies you mostly tend to work with and consult for. Typically higher on their priority list is moving the image registry into their cluster. People are pretty ok with building with other providers like Gitlab, AWS, Azure. So for me the best value of this video was the explaination of what it REALLY means "Kubernetes is deprecating Docker " - how it's not dramatic at all. Docker doesn't build "Docker images" as their is no such thing - but rather OCI-compliant images. Brilliant. All in all: informative, valuable - and entertaining - as always Viktor, keep it up :)

[@DevOpsToolkit]

It all depends on the state of a company and what would benefit more from "modernization".

[@typeer]

Thanks very much learned heaps

[@DevOpsToolkit]

Glad to hear it!

[@gatsbylee2773]

Thank you.

[@DevOpsToolkit]

You're welcome

[@ruslanivanov831]

"docker is caput" ha-ha ))) it's made my day )))

[@opensourceguy730]

Software-2-Image, or s2i, is another great option. Podman Desktop is another option for your desktop, too.

[@manofacertainrage856]

Kaniko will fail to build some Dockerfiles that Docker builds - like when trying to install gnupg2 to a recent (12) Debian image. I think Docker is automatically answering a config question and continuing while Kaniko kicks me out to an alternatives prompt. I haven't found a way to get around this issue yet. I have used Kaniko for other cases though.

[@andreykaliazin4852]

Hi Victor! Sorry about the deleted comment. I was not aware of the youtube's habit to remove comments without a warning. I was only asking to review using buildah as the image building utility, possibly in comparison to kaniko. We do use rhel8 variants on prem and in the cloud, so it would be natural to use buildah in k8s too.

[@andreykaliazin4852]

@@DevOpsToolkit Thanks a lot and sorry for the typo in your name, Viktor!

[@jaysistar2711]

Thanks for this information! I've been needing a way to build images from a very hetrogenous cluster (Linux ARM64, Linux AMD64, Windows AMD64). I'll try Kaniko and see how it goes.

[@DevOpsToolkit]

I'm curious how it turned out for you and whether Kaniko fits your use cases. Please let me know how it went.

[@cooljai565656]

Hey please share your experience

[@jaysistar2711]

@@cooljai565656 It didn't go well. I could only build images that are single stage. Kaniko removes critical files in the image when going to a new stage, and doesn't replace them. I went down the rabbit hole of moving those files, making symlinks to them, and attempting to replace symlinks at the beginning of every stage, but that required modifying Dockerfiles to make it work. Any app that uses glibc will fail to start in the second stage, so I also switched everything to statically link MUSL instead. It turned out to be too much work, and too much had to change to accommodate Kaniko. I'll try it again, if these problems are fixed somehow, but right now its unusable for me, and I have to have a separate build agent from my k8s cluster. I still have another k8s cluster with Docker Engine that works fine, as well, but it builds with Docker Engine, not Kaniko. In addition to those problems, with Kaniko I could only build the Linux containers, not the Windows containers.

[@patricknelson]

Since nobody has commented on it yet... your shirt is hilarious!

[@DevOpsToolkit]

Thanks :)

[@anishkumaryadav2636]

But what about multi-arch images? We can create multi-arch images and push via Kaniko but there is still no support for manifests list and docker buildx? Any help is highly welcomed

[@DevOpsToolkit]

Something similar to the manifest list is currently not supported in Kaniko, and I'm not sure it ever will. The logic is that Kaniko builds a single image from inside a single container. You would need to run it in a loop (preferably in parallel) to get build-arch images.

[@techsolutions123]

great explanation ,thank you ,curious whats your webcam,software setup or you had a video on how you doing this thanks

[@DevOpsToolkit]

I have last year's iMac that has fantastic camera build in (much better than any other webcam). On top of that, I also have Sony ZV1 camera. As for software, I record with OBS and edit with final cut pro. Lights and mic are from ElGato.

[@xlisaable]

Thanks for the video. I have a question: I have a Kubernetes executor in my gitlab CI/CD pipeline. Could I use kaniko to build a docker file or is that also not possible?

[@DevOpsToolkit]

Yes you can. Kaniko is a binary and as long as you can execute it (in or out of a container), you can build container images based on Dockerfile.

[@deepanshumishra4568]

If I want to use local context directory and dockerfile path do I need to omit workspace with my working directory? and what will be the arguments? in --dockerfile=? and --context=? please reply

[@DevOpsToolkit]

I always specified those things explicitly so I'm not 100% sure what are the default. I'm guessing that `--dockerfile` defaults to `Dockerfile` in the current dir and `--context` to the current dir as well. Just bear in mind that "local" is local to the context of the container where it's running.

[@ritesharyal2815]

liked it

[@estudiotademan444]

Amazing ! There is a thing i dont understand. Could you give me a couple of reasons or scenarios where we need to create images inside a Kubernetes Cluster ? We usually do it from Gitlab CI. Thanks in advance!!

[@DevOpsToolkit]

Normally, when people start transitioning to Kubernetes, they move their pipelines there as well. So, if pipelines are running in Kubernetes, typically with each step being a different container, you need a way to build images from containers. GitLab CI has a Kubernetes runner and, if you start using it, you'll have to be able to do all the steps in Kubernetes, including building container images.

[@estudiotademan444]

@@DevOpsToolkit Genious.

[@javisartdesign]

I dunno if you can use DinD using kubernetes without docker being the container engine.

[@ronaldm.1556]

yes you can. Dind works very well and does not need to connect to the host's socket.

[@sanketsingh1683]

Thanks for the video. very helpful. Is there a way to perform build docker image and push docker image to image repository in separate commands using Kaniko? I want to build image which pull base image from an ECR (xyz) and then creates the build and then push to application image to another ECR (abc) so I need to authenticate to different ECR's and perform build and push in separate phase of gitlab.

[@DevOpsToolkit]

I haven't tried that so I'm not sure. I always tend to keep images in the same registry.

[@alex.prodigy]

i'm planning to migrate to kaniko on gitlab-ci pipelines but for now we are just using dind for docker daemon and docker client communication i guess you can do that in k8s as well , even if docker shim is removed or you use another container runtime for your k8s cluster

[@DevOpsToolkit]

I think it might work. I never tried running a k8s cluster with two container runtimes (one for k8s and Docker) so I cannot say that for sure. In situations when building had to go through Docker, my preference was to have Docker in a VM instead of having it inside k8s.

[@alex.prodigy]

@@DevOpsToolkit i don't see any reason why it would not work , it's just another service after all ... even if it's the docker daemon in this case. But one thing that i really don't like the dind with gitlab-ci is that the host/server docker daemon needs to be in privileged mode

[@DevOpsToolkit]

@@alex.prodigy It probably works. I'm not saying that it will certainly work only because I haven't tried it. Privileged mode is certainly the issue. Another potential issue might be resources. If Kubernetes is unaware that Docker is spinning up containers, it might not have the correct picture about the available resources when scheduling Pods. Again, just as with the previous comment, it's a wild guess on my part since I haven't tried it.

[@alex.prodigy]

@@DevOpsToolkit you are right , my comment is theoretical since i haven't tried it either :)) Anyhow , thank you for making these videos , are quite helpful in getting to know some technology or tool or for awareness

[@yassinemessaoud8619]

Thank you for this tutorial but i'm a bit confused: -Docker uses a docker host (engine) to deploy many instances of containers what host does Kaniko uses ? -Relating to first question, when using Kubernetes with Kaniko does Kubernetes manages different Kaniko hosts so if one host fails we can migrate to the other hosts as in Docker or is there another strategy ?

[@DevOpsToolkit]

You are mixing the different purposes of Docker. We can use it to run containers and to build container images from which containers are made. Containers can run through any container runtime engine. That could be Docker (but not in Kubernetes), it could be ContainerD (container engine that is baked in Docker but also runs as standalone), cri-o, etc. When we want to run containers, we are basically saying "here's an image, run it". Kaniko has nothing to do with how we run containers, but how we build containers. It replaces `docker image build` and NOT `docker container run` or `kubectl apply` and similar commands. All in all, Kaniko is used only to build images, and NOT to run containers.

[@yassinemessaoud8619]

@@DevOpsToolkit So after building an image with kaniko does K8s take care of running containers of this image ?

[@DevOpsToolkit]

Yes. Those are completely separate processes. You run containers based on images built by you or someone else.

[@yassinemessaoud8619]

@@DevOpsToolkit Thanks a lot !

[@vinayhegde9888]

Anyway to implement this on eks-faregate!!? Need to implement github action runner on eks-faregate and it isn’t supported Dind. Any thoughts!!?? Plz respond.

[@DevOpsToolkit]

Kaniko runs inside containers without any special requirements so i don't see a reason why it wouldn't work in EKS Fargate. I do not use Fargate myself so I might be wrong though.

[@chandup]

Nice video. Thank you for this. How to build multi stage container images using kaniko?

[@DevOpsToolkit]

That works just fine without any additional effort. As long as there is Dockerfile, Kaniko does not care much whether it is a single or multi-state. I used github.com/vfarcic/kaniko-demo/blob/master/Dockerfile in the demo and, as you can see, there are 2 stages in that Dockefile.

[@yassinemessaoud8619]

Can i use minikube to run the images built by Kaniko instead of K3D ?

[@DevOpsToolkit]

Kaniko builds the same container images as any other tool (e.g., Docker). As a result, you can run them (convert images into containers) with any container runtime, which means that you can use them inside any Kubernetes cluster, including minikube.

[@technically1055]

Thanks bro 🎉

[@bgroesser]

You can still use docker build. K8 uses a different runtime but you can still use docker to do the build.

[@DevOpsToolkit]

You can, but only if that build is not happening inside Kubernetes. Given that more and more tools are moving into Kubernetes, that means that the usefulness of Docker is going down. Good examples are self-managed pipeline solutions (what many call CI/CD). Most of them are running in Kubernetes or, if they're not, the agents they're using are in Kubernetes. Given that almost everyone builds containers as one of their pipeline (CI/CD) steps, that means that Docker is a no-go. So yes. You're right. You can still use Docker to build, but only if you are not building inside Kubernetes. Since automation is moving into Kubernetes, that means that you can build with Docker manually or through automation tools that are not running in k8s.

[@marsel4717]

@@DevOpsToolkit You can still pull images or build them using docker build command inside the k8s, they only needed to be pushed to some registry to allow them to be used by Kubernetes

[@robinhughes-jones4034]

"docker socket shenanigans"

[@creative-commons-videos]

So if i am using CI/CD tool like Gibhub Actions, CodePipeline etc to build container image, do i have to change that from docker to kaniko ?

[@DevOpsToolkit]

If you are using GitHub Actions, you are not running your pipelines (actions) in Kubernetes. You are running them in VMs that have Docker installed. As such, DockerShim is not used, you are not mounting sockets, and you are not running pods in privileged mode. You can keep building images using Docker :) A separate discussion would be on the topic of GitHub Actions missing the opportunities presented when running in Kubernetes. But, the long story short, Docker is fine in GitHub Actions.

[@mbesida]

@@DevOpsToolkit What's the point in building app images inside kubernetes? If you have some CI/CD tool as already mentioned Github Actions you can build an image with docker directly and push to whatever registry. Once you have application environments(like dev and prod) for your service, isn't a bad idea also to have builds on the same kubernetes cluster? And even this is different clsuter, why to spin up additional one if CI/CD tool can do that?

[@DevOpsToolkit]

When using Saas (like GitGHub Actions) it does not matter whether it is Docker or something else. A service provider is handling it and you are just saying "I want to build an image, it's your job to figure out how". It's a similar situation as with, let's say, Google Cloud Run. Is it Kubernetes or not behind it? It does not matter. It's not something a user cares abound. It's an implementation detail that is hidden. The above comments are valid only if you use GH Actions fully as SaaS. If you do start using agents that you run yourself, the story is different. From that moment on, it's not "not my problem" any more. When I'm running it, it matters whether it is Docker or something else. Docker does not scale, Docker daemon cannot run in a container, etc.

[@mbesida]

@@DevOpsToolkit thanks for the answer, did a small research on a matter... Now I finally understood what are possible options for ci/cd and how useful the approach described in your video can be.

[@stevenalves7506]

I found this video very interesting but I still confused... Why I should build container images from a Dockerfile, inside a container or Kubernetes cluster and not locally? If I build I Image with docker I'm not be able to run it in a k8s cluster with other image container runtime like containerd?

[@DevOpsToolkit]

It does not matter how you build an image, they are all following the same standard and they can run in k8s. The reason why you shouldn't build locally lies in automation. You want to automate repetitive tasks, typically whenever you push changes to git repos. Build binaries, run unit tests, build images, run functional and integration tests, deploy to staging, deploy to production, etc. All those things are repetitive and can take a while to execute. There is no good reason why would you "waste" your time on those, instead of, let's say, write code. Now, if all the repetitive tasks are automated, they are likely going to be executed in Kubernetes clusters. Hence the needs to do everything inside containers, including the task of building container images.

[@maylingtan3716]

Thank you Viktor. May I ask if this can work with bitbucket repo? I can get it working with private Github repo but not with Bitbucket. Thank you.

[@DevOpsToolkit]

If you're referring to the build context, I think it works only with GitHub (github.com/GoogleContainerTools/kaniko#using-private-git-repository). However, that shouldn't be a probably since you should be able to use local filesystem as the build context. Assuming that you're running Kaniko as part of a pipeline build (Jenkins, Tekton, Argo Workflows, etc.), you are probably already cloning the code anyways.

[@maylingtan3716]

​@@DevOpsToolkit , I am using Argo Workflows, do you have a template or reference I can refer to, on how to clone it in argo workflow and set the context to the cloned volume? Thank you.

[@DevOpsToolkit]

I don't have an example at hand :(. What you need to do is clone the code and than moved it through a volume from one step to another.

[@suhchuu3603]

So... we're still using the docker bindings but maybe in a different way? What we do is actually have a container within the pod which is running the DIND docker engine. So what we do is basically have the socket binding on a "main" container, which links it to the "dind" sidecar. What is your take on this type of configuration? We don't solely do this for building / uploading images - we use it to run containers for things like testing frameworks. Is there a better way to accomplish something like this? I admit it is a bit gross, and the docker dind needs to run with privelages so it sounds like something nice to toss out. Maybe I should instead be looking into different container runtimes and trying to run it directly from the "main" container itself? Love your videos btw, amazing content!

[@DevOpsToolkit]

I am guessing (correct me if I'm wrong) that you run containers with docker and not inside kubernetes. If that is the case, I suggest switching to Kubernetes One of the big advantages of Kubernetes is that it orchestrates containers across a group of clusters. However, docker is not supported in Kubernetes any more (there is a way to use it but I'll ignore that). That means that things like DinD do not work simply because there is no D in k8s.

[@Ruben-by4oy]

I thought docker deprecation means that docker will not be the primary runtime for running containers. It means docker will stay, but will be equal to containerd or others.

[@DevOpsToolkit]

No. Docker is removed from Kubernetes codebase as well as from clusters since it does not implement OCI.

[@julianomoraisbarbosa]

👌👏

[@comworkio]

After struggling with the slow of Kaniko and the cache problems, we've chosen buildkit daemon-less instead which is perfectly fine running on a K8S pod (like a tekton task for example).

[@lextr3110]

do you know if these problems was resolve in current kaniko release?

[@sasankv9919]

What if it's a private source git repo. How do we work with that

[@DevOpsToolkit]

You can do something like github.com/GoogleContainerTools/kaniko#using-private-git-repository

[@faller222]

I have my gitlab pipeline creating containers, does gitlab use containers, what are the runners?

[@DevOpsToolkit]

GitLab runners can run on VMs or in containers so it all depends on the way you set it up. P.S. I haven't used GitLab CI enough to go into a deeper conversation about it. If that'll help, I can add it to my TODO list and get back to you once I switch from GitLab newbie to the GitLab CI "master" :)

[@faller222]

@@DevOpsToolkit dont worry, thanks for the response

[@Flyingnobull]

@@DevOpsToolkit Gitlab CI is great. You have the options to do it on the shell, docker or Kubernetes. And some others. What I can say is the way you used docker in another docker resembles gitlab builds in docker containers. Gitlab also has to mount docker socket on the container too. So I'm assuming same security issues apply.

[@DevOpsToolkit]

@@Flyingnobull Nothing really prevents you from using any other tool to build container images through GitLabCI. It can run any container and that means that you can switch to Kaniko or anything else. All pipeline tools (GitLabCI included) are essentially a way to execute some steps in different machines, containers, etc.

[@lamnot.]

So how can we build immutable build artifacts with Kaniko?

[@DevOpsToolkit]

It builds OCI images that are immutable.

[@DevOpsToolkit]

Posting a message from Jeff Ferber (KZbin deleted his comments since they container links): This video doesn't cover some of Kaniko's downsides. It is not concurrency safe and each container should be torn down after each run. Details in issue 1869 of their github repo: GoogleContainerTools kaniko. Also, kaniko will not execute in most other base images, so it would be difficult or impossible to use in a FaaS environment like Lambda or Knative. More info: github.com/GoogleContainerTools/kaniko/issues/1869

[@lextr3110]

so what is your perfectly working alternative option? was this fixed?

[@DevOpsToolkit]

Concurrency should not be a problem if you are running kaniko in one-shot containers. Also, for me, having to tear down containers after each build is not an issue. Kaniko is often used inside pipeline that do just that (spin up container(s) for every execution/build).

[@iamkirankumarkv]

if we don't give registry to push, where will it store the image?

[@DevOpsToolkit]

Typically, image name is the full path to the registry. That's where it stores it.

[@creative-commons-videos]

Hi there, thanks for such informative video, i tried to follow the exact step of the video, everything worked well except at last it hangs "INFO[0270] Taking snapshot of full filesystem..." and then it give error ContainerReady=false

[@DevOpsToolkit]

I haven't experienced that error. If that helps, we can have a screen-sharing session and try to figure out what's wrong together.

[@creative-commons-videos]

@@DevOpsToolkit That would be my honour to have you but i am not a devops guy, i am just a simple software developer, just started learning devops, mostly watching your videos, so there will be no help from my side for debugging as i know nothing about these things, hope it's fine :)

[@DevOpsToolkit]

@@creative-commons-videos I mean that if I can see your screen, I might be able to figure out what's wrong in your case. It would be a private session :)

[@creative-commons-videos]

@@DevOpsToolkit sure why not

[@creative-commons-videos]

should i send you zoom / google meet link ???

[@bules12]

⚠ WARNING ⚠ In my case, Kaniko was not very fast building images. In a contabo server VPS with ubuntu 20.04 8GB ram it takes 3 minutes to build. With kaniko, 15minutes. It was a NextJs image using T3 stack If you doesn't need to perform a CI/CD in various projects concurrently, I would recommend you to create an script to connect through ssh to a server, build the images and push them to your registry Cheers

[@DevOpsToolkit]

That difference is probably due to using cache from the server when building without Kaniko and not mounting the same cache into Kaniko container.

[@bules12]

@@DevOpsToolkit I did both operations with no cache 🤔 the contabo server was brand newly created

[@DevOpsToolkit]

Hmmm... I haven't experienced such a big difference so I'm guessing you hit a "special" condition. It would be great to open an issue in the Kaniko project so that they, hopefully, fix it.

[@bules12]

@@DevOpsToolkit there are some issues with the same problem, I tried using -cache, -cache-dir and -snapshotMode=redo, it helped reducing from 15 to 8-10 minutes

[@underlecht]

Maybe we can build docker containers using dind (docker-in-docker) image?

[@DevOpsToolkit]

Docker in Docker (DinD) does not mean that Docker is fully running inside containers. You need to mount a volume to a Docker socket running on the node or to run containers in privileged mode.

[@hyunilshin9944]

@@DevOpsToolkit I'm curious. Without docker support in Kubernetes, can we still use DinD?

[@DevOpsToolkit]

@@hyunilshin9944 I do not think so. Even if we could, we shouldn't. DinD was never a good idea. In the past, it was necessary evil. Since then, Kubernetes and containers evolved and it would need to be a very special case to justify DinD. Can you tell me what you're trying to do with DinD? I might have a suggestion how to accomplish the same (or better) without it.

[@JaydeepDave12]

brew install bat (cat alternative)

[@DevOpsToolkit]

The problem is that I'm trying to make it work on the widest number of platforms/OSes and with a minimal effort for others. That's why, for example, I use Bash in videos even though I'm using Zsh in all other scenarios. I'm afraid of using anything "extra" that would complicate people's lives when following the demos.

[@andreykaliazin4852]

@@DevOpsToolkit Could you please make a video on how Zsh is better than Bash for you? everyday tips, hints and just fun with zsh - anything useful (apart from oh-my-zsh). Thanks in advance!

[@smerlos]

bye bye "docker build" ....