[ Kube 69 ] Using Private Docker Registry in Kubernetes

[@peliusrex2730]

Awesome tutorial. Thanks for walking through common errors. Most tutorials just show the right way to do stuff without showing the common pitfalls.

[@justmeandopensource]

Hi Pelius, thanks for watching.

[@ulrichoke9639]

Very clear, All three scenarios covered simply in a short time. Interesting !

[@justmeandopensource]

Hi Ulrich, thanks for watching.

[@FabulousMindset]

Great information, thanks! I like a lot your style, is like watching a live demo but ... on demand :D I see you are very good with shells already, here are two things I feel sharing for the benefits of everyone reading: CTRL+l (lower letter "L") works like typing the command "clear" ALT+. pastes the last arg of the previous command line All the best and keep up with the good work! And, and almost forgot :) I loved the background at the beginning, Namaste brother _/\_ ; )

[@justmeandopensource]

Hi Fabio, many thanks for your comment. Its hard for me erase my muscle memory about "clear". Reaching CTRL involves moving my hand little bit down. I use ESC+. for your ALT+. suggestion. Namaste and thanks for watching.

[@sahaniarunitm]

your video once again helped me for solving insecure issue from registry.Great

[@justmeandopensource]

Hi Arun, many thanks for watching. Glad it helped. Cheers.

[@sahaniarunitm]

@@justmeandopensource we are looking ahead for your new new video.

[@justmeandopensource]

@@sahaniarunitm Thanks.

[@taharbounsiar137]

Hi Vankat, thanks for this great work, im learning a lot from your videos about k8s.

[@justmeandopensource]

Hi Tahar, many thanks for your interest in my channel. Cheers.

[@leonardogodoi533]

You helped me a lot! I was really struggling to create a registry. Thank you!

[@justmeandopensource]

Hi Leonardo, thanks for watching. Glad it was helpful. Cheers

[@mustufamithaiwala7205]

Awesome Venkat, Good Going!!!

[@justmeandopensource]

Thanks Mustufa for watching. Cheers.

[@Pyrusmalus]

Thanks is not enough the ways to taught.. shukriya❤️

[@justmeandopensource]

Hi Adil, many thanks for your interest in my videos. Cheers.

[@ramkumarmusulla9113]

Nice Demonstration. Thanks Venkat.

[@justmeandopensource]

Hi Ram, thanks for watching. Cheers.

[@tan3ryigit]

Very good tutorial indeed

[@justmeandopensource]

Hi Taner, thanks for watching.

[@kausikdas9664]

Hi, Thank you for the awesome tutorial ......in the 11:26 minute of the video you are add insecure registries in the worker1 and 2 nodes to solve the https issue, i am unable to get the ips of the worker nodes and unable to to do ssh with root@worker1 &2 , it says "ssh: Could not resolve hostname worker1: Temporary failure in name resolution".....please help

[@justmeandopensource]

Hi Kausik, thanks for watching. Are all the machines virtual machines? Why can't you get the IP addresses for them?

[@kiranpadam5703]

very good venkat,thanks

[@justmeandopensource]

Hi Kiran, thanks for watching.

[@AnandKumar-dc2bf]

Simply awesome...

[@justmeandopensource]

Hi Anand, thanks for watching.

[@abhijeetsamanta4246]

Hi Venkat, I am getting error while push the image using docker push. error : "The push refers to repository [localhost:8000/dfarm-app1] Get localhost:8000/v2/: read tcp 127.0.0.1:56584->127.0.0.1:8000: read: connection reset by peer". Could you please help me to fix the issue?

[@justmeandopensource]

Hi Abhijeet, thanks for watching. Have you added the insecure-registries to the docker config? Worth adding the config and trying it again. Thanks.

[@yamen1150]

Thank you for the tutorial.. I wonder why you didn't need to add the "domain.crt" file in your kubernetes cluster hosts! How did the local private registry with an enabled TLS accept the connection without public key (the CRT file in this case)? Could it be because the docker was still configure to accept your registry as insecure, therefore the TLS was bypassed? I have a k8s cluster created by kubeadm and I'm not able to pull images through kubectl while I can through docker. The error "x509: certificate signed by unknown authority" haunts me down when I create a pod, eventhough I have everything configured properly with the CRTs added.

[@ylcnky9406]

Hi Venkat, today I finally completed the Kubernetes series in your playlist. Great great great thanks. Are you planning to cover deployment and usage of Kafka and Apache Spark with Kubernetes? If not can you please consider this as a request for future videos?

[@justmeandopensource]

Wow. You managed to watch all the videos. Many thanks. My focus at the moment is on kubernetes, AWS and MongoDB. I am not entirely sure whether I will have time to do Kafka and Spark. I haven't used either of those. So that will be something I need to learn first. Cheers.

[@konstantinosraftopoulos9918]

I created the docker registry in my kubernetes master vm. I followed all the steps that you described i your video and more important i filled in my worker node vm in the /etc/docker/daemon.json the "insecure-registries": ["ip_of_master_node" : 5000] but the error http: server gave HTTP response to HTTPS client continue to appear. :/ **Should i fill also the daemon.json file woth the "insecure-registries"?

[@Youdude2]

can you do a video on harbor for registry?

[@user-rp9iis1en6h]

Thanks man. can we expose this local registry as https using nginx? if yes than how? or what is the best way to expose it as https service so that other kubernetes cluster can access it over the internet as a secured registry?

[@ashwathmendan732]

Thank you so much sir

[@justmeandopensource]

HI Ashwath, thanks for watching.

[@iknownothing8949]

What are the advantages of using private registry vs own repo/account at Docker Hub.

[@justmeandopensource]

You host the repository yourself so everything is under your control. Don't have to rely on cloud or any third party provider and you don't have to trust anyone for storing your images securely.

[@iknownothing8949]

@@justmeandopensource Thank you. 👍

[@justmeandopensource]

@@iknownothing8949 You are welcome. Cheers.

[@vaporlander2621]

Great - Thanks a lot

[@justmeandopensource]

Thanks for watching.

[@sumitneg1984]

how to pull the docker registry on localhost if the container run time engine is containerd in the kubernetes worker nodes. The Https error is coming while pulling the image

[@justmeandopensource]

Hi Sumit, thanks for watching. Can you paste the actual error message?

[@sumitneg1984]

@@justmeandopensource "server gave http response to https client" some of the action that i took are as under:- 1. updated Toml config file of containerd [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = ["registry-1.docker.io"] [plugin."io.containerd.grpc.v1.cri".registry.mirrors."test.registry"] endpoint = ["test.registry:5000"] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.configs."test.registry".tls] insecure_skip_verify = true where test.registry is the domain name of my local repository whaich is running on port 5000. It did not work still the same error was coming . 2. second, step was that I binded the local registry with self sign certificate which were mounted in the volume of local repository container. SO now it is working on port 443 (ie test.registry). now the privious error is gone but it is still not pulling image. logs state that image cant be pulled. pl help

[@sumitneg1984]

and the error is that x.509 certificate cannot be trusted

[@ashishjohn9708]

Hi Venkat, in docker hub when we select public repo instead of private i am still unable to pull image in kubernetes cluster why?

[@justmeandopensource]

Hi Ashish, thanks for watching. You shouldn't be having any problems pulling from public docker repo. May be if you post the errors it will be useful. Cheers.

[@chaitanyayadav1889]

Hi, Can you suggest that which is of reasonable cost Docker private registry or Azure container registry ?

[@justmeandopensource]

Hi Chaitanya, thanks for watching. I have no idea about the cost as I haven't used any of those.

[@dmitryparkhomenko5009]

Hi Venkat, how to install docker registry ui on the k8s, sometimes it comfortable manage docker images

[@justmeandopensource]

Hi Dmitry, thanks for watching. Do you have any particular UI in mind? I can give it a try.

[@vijaykaushik9080]

Great Video! But i am still not clear where this private registry resides. Is in K8s cluster or another standalone server/deployment server ?

[@justmeandopensource]

Thanks for watching. If you look at the notes I shown at the start of the video, you will see that the private docker registry is running on my host linux machine outside of the kubernetes cluster. This video is just to demonstrate how to connect to a private docker registry for pulling container images.

[@harshgoyal6822]

Very clear and simple way to explain things. Can you pls let me know is there any platform when i can practice this stuff like production level? Because right now i am working in QA and i dont have exposure to devops. But i want to get into Kubernetes and Devops

[@justmeandopensource]

Hi Harsh, thanks for watching. You can always practise this in your laptop as most of my videos are bare-metal based and not cloud based. Or you can get yourself a free Google Cloud account and use the free credits for 12 months. kzbin.info/aero/PL34sAs7_26wM4ETdsdrRrO1euoUtO4jF6 There are lots of ways to start your learning adventure.

[@harshgoyal6822]

@@justmeandopensource Hi Thanks for replying. Yes i started using AWS. Actually my target is CKAD or CKA certification. Can you pls provide me some content or pdf book etc which can help me? THanks in advance!

[@TheJrocksk8]

Hello, What does the curly braces represent in the end of the yaml file? And how does it relate with the path in the local host? Here is a part of the JSON code. .... volumes: - docker-registry-data:/var/lib/registry volumes: docker-registry-data: {}

[@justmeandopensource]

Hi Jorge, thanks for watching. Its a notation for docker volumes. We are not mounting any existing directory inside the container but creating docker volumes. volumes: docker-registry-data: {} is same as volumes: docker-registry-data: We are just specifying that no options are passed to this volume. Just a docker volume will get created with that name and it will reside under /var/lib/docker/volumes. Thanks,.

[@gbrt9569]

Hi Venkat, slightly off topic, but what are you using for the that ghost shell completion? thats pretty cool !! is it something like 'powerline'?...also the status bar at the bottom of your screeen showing your IP etc, what is that?

[@justmeandopensource]

HI Gary, thanks for watching. I am using Mnajaro with I3 Tiling Window Manager. For shell, I use ZSh + Oh-my-zsh + zsh-autosuggestion on Gnome Terminal. It is zsh-autosuggestion plugin that auto suggests commands from my history. github.com/ohmyzsh/ohmyzsh github.com/zsh-users/zsh-autosuggestions I have also done few videos on My customization and got some Ansible playbooks to automate the setup. The playlist is at kzbin.info/aero/PL34sAs7_26wOgqJAHey16337dkqahonNX Thanks.

[@gbrt9569]

Thats great, thanks for that :) .. i dont suppose you could share some of the config in your zshrc file? or is it pretty standard?

[@justmeandopensource]

@@gbrt9569 My zshrc is pretty standard one. I have a separate .zshrc-venkatn where I have my aliases/exports/functions and I source it in .zshrc. Only other thing I have updated in my standard zshrc is added zsh-autosuggestions, git, kubectl to the plugins list.

[@TheSibaSwain]

Hi Venkat,is it possible to setup a UI for the registry to browse through all the repository and images. it will be great if you can post a tutorial on this. Thank you

[@justmeandopensource]

Hi Siba, thanks for watching. There are lots of Github projects for this. A quick search gave me this one. hub.docker.com/r/joxit/docker-registry-ui/

[@TheSibaSwain]

@@justmeandopensource Thanks venkat for your reply. I am using the below container to setup one,but seems misiing something. registry.hub.docker.com/r/konradkleine/docker-registry-frontend This is my compose file: version: '3' services: docker-registry: container_name: docker-registry image: registry:2 ports: - 5000:5000 restart: always environment: REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt REGISTRY_HTTP_TLS_KEY: /certs/domain.key REGISTRY_AUTH: htpasswd REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm volumes: - ./registry-data:/var/lib/registry - ./certs:/certs - ./auth:/auth docker-registry-ui: container_name: docker-registry-ui image: konradkleine/docker-registry-frontend:v2 ports: - 8080:80 environment: ENV_DOCKER_REGISTRY_HOST: docker-registry ENV_DOCKER_REGISTRY_PORT: 5000 I can push image to registry but in UI nothing showing up

[@nikhilrajan2370]

Can we create a private docker registry and push and pull from it from an external docker client

[@justmeandopensource]

Hi Nikhil, yes you can. All you need is a docker registry accessible to clients and credentials if it is password protected.

[@nikhilrajan2370]

@@justmeandopensource ok so i am going to set this up on the master node of my unmanaged on prem kubernetes , mostly because of a lack of hardware . So to configure it as a insecure registry , i would have to do a docker restart , would that mess up my cluster

[@justmeandopensource]

@@nikhilrajan2370 Yes, after editing /etc/docker/daemon.json, you need to restart docker service which will cause an interruption to all the running docker containers and you will have a very brief downtime of few seconds where your cluster won't be accessible.

[@prabhasrockstar]

I setup docker registry that deploy on k8s cluster but I getting error ssl certificate issue

[@zhang20244]

Very cool, could you show how to install Kubeflow on k8s ？ that is machine learning platform, and how to config ISTIO for micro services gateway

[@justmeandopensource]

Hi Frank, thanks for watching. I haven't explored Kubeflow yet. If I get some time I will have a play with it and decide if I can make a video. Cheers.

[@zhang20244]

Thank you so much, your video very easy to understand, I like all of your training video

[@justmeandopensource]

@@zhang20244 You are welcome. Cheers.

[@8802148204]

Is daemon.json still required on workers when I use TLS certs for docker-registry in 2nd approach? I just don't want to make this change on all worker hosts.

[@iknownothing8949]

Hello, two question, followed the video and all was success. The first issue is when trying to building/tagging and pushing an image from local computer to private registry on the server (linode) - i get no errors (all seems to work fine), but I cannot see the images (I am logged in via docker login etc.) so then I tried doing a curl -k localhost:5000/v2/_catalog => Received HTTP/0.9 when not allowed when doing: curl -k localhost:5000/v2/_catalog (tried the ip of the node as well) => {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail" and tried passing the --cert and --key - and same error. Any hints would be appreciated.

[@stevecorbin9102]

Hi Venkat, I am getting doesn't contain any IP SANs error. When you use the secure docker registry do you still need the daemon.json file?

[@justmeandopensource]

Hi Steve, thanks for watching. It depends on the way you create the certificate. The usual practice is to use subject alternate names SAN while generating the certificate. For example, if you are generating a certificate for docker-registry.example.com, then you would also want to include docker-registry (short name) in SAN in certificate. If you don't include alias (SANs), then you will have to make sure to access the server using the same name given in the certificate (docker-registry.example.com) and when you use just the shortname docker-registry, you will have certificate error. So you either update the certificate accordingly or use whatever cn (common name) you given in the certificate for the docker registry or add an entry for insecure registry in daemon.json. Hope it makes sense.

[@MohitVerma-yj1pq]

add subjectAltName = "IP:192.x.x.x" in openssl config file

[@nikhilrajan2370]

Also i am having to use sudo , to access docker .. can i bypass this ..without adding the user explicitly to docker user

[@justmeandopensource]

Hi Nikhil, thanks for watching. There is only two way. Either you run docker commands with sudo or add your user to docker group.

[@dirbacke]

Thank you for your great tutorials. I was wondering if you could help me with a tricky question. I followed one of you other tutorials to setup Jenkins using Helm. And now I would love for Jenkins to get the images from the Docker registry that I have setup using this tutorial. But all I get from Jenkins logs is "x509: certificate signed by unknown authority". I have changed the openssl.cnf to contain [alt_names] IP = "172.x.x.x" # the IP address of my docker registry on a private cloud server. What can I do to get the images to Jenkins using my private Docker registry? I am grateful for any help.

[@SonNguyen-pw8lm]

Hello bro, How to remove image in docker registry? Thanks bro

[@myvillagemyfriends7306]

Master Super

[@justmeandopensource]

Many thanks for watching this video. Cheers.

[@anilprajapati4922]

After adding the insecure-registries to the docker config, till getting the error in the kubectl events, Failed to pull image "10.6.0.95:5000/nginxprivate": rpc error: code = Unknown desc = Error response from daemon: Get 10.6.0.95:5000/v2/: http: server gave HTTP response to HTTPS client.. can you help me ?

[@justmeandopensource]

Hi Anil, thanks for watching. What exactly you added in docker daemon.json file? It seems you haven't added the right url for docker registry or you haven't restarted the docker service on the worker nodes.

[@konstantinosraftopoulos9918]

@@justmeandopensource i have the same problem. I created the docker registry in my master node and in my worker node in the daemon.json i provided the ip of the master, master_ip:5000, but doesnt work..

[@prakharkurele3061]

Brother I have deploy docker registry as node port and it will work on localhost:portnumber from all the nodes. Is this right.

[@justmeandopensource]

Hi Prakhar, yes it should be accessible within all nodes using localhost:. Thanks.

[@zerobig5517]

Hello, my friend, Thank you for posting useful and informative content every time. I have a question about the private container registry. Do you think it's better to build and use the container registry ourselves even in a sensitive customer service environment with some traffic and an increasing trend? For your information, we are planning to use Managed Kubernetes, but the container registry in the cloud is not allowed. Do you have any additional options or recommendations to consider in this situation?

[@abhijeetsamanta4246]

H I Venkat, Thanks for this awesome tutorial which I was looking, however If possible you help me how I can make the docker image in private of Node.js based backed and use in k8s as pods?

[@justmeandopensource]

Hi Abhijeet, thanks for watching. You will have to do docker login to login to your private docker registry. Then build and push the image to this private registry. Other steps should be similar as shown in this video. Cheers.

[@stocks-fox]

Failed to pull image "localhost:5000/ubuntu": rpc error: code = Unknown desc = Error response from daemon: Get localhost:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused

[@justmeandopensource]

Hi Ahmed, where is your private docker registry running? You are using localhost:5000/ubuntu as the url which won't work inside the worker node as docker registry may not be running there. Please change localhost to ip address of the machine where you are running the docker registry.

[@stocks-fox]

@@justmeandopensource Failed to pull image "192.168.39.1:5000/my-ubuntu": rpc error: code = Unknown desc = Error response from daemon: Get 192.168.39.1:5000/v2/: http: server gave HTTP response to HTTPS client

[@stocks-fox]

and i added "insecure-registries" : ["myregistrydomain.com:5000"]

[@muradheyderov]

👏👏👏

[@justmeandopensource]

Hi Murad, thanks for watching. Cheers.

[@lakshmicheran6341]

hi we are still facing imagepullback

[@diegogarciaarriaza5754]

Hey friend, thank you for the tutorial, very clean and clear But I need your help, when I try to deploy with kubectl run mytest ... y get this: "ImagePullBackOff" And if we see the logs.... " Normal Scheduled 54s default-scheduler Successfully assigned default/mytest to kunode1 Normal BackOff 23s (x2 over 53s) kubelet, kunode1 Back-off pulling image "localhost:5000/myalpine:v1" Warning Failed 23s (x2 over 53s) kubelet, kunode1 Error: ImagePullBackOff Normal Pulling 8s (x3 over 53s) kubelet, kunode1 Pulling image "localhost:5000/myalpine:v1" Warning Failed 8s (x3 over 53s) kubelet, kunode1 Failed to pull image "localhost:5000/myalpine:v1": rpc error: code = Unknown desc = Error response from daemon: Get localhost:5000/v2/: dial tcp [::1]:5000: connect: connection refused Warning Failed 8s (x3 over 53s) kubelet, kunode1 Error: ErrImagePull" Is this normal?, I have try with user/pass registry:2, setting secrets, etc... And yes, I have configured the registry ok and I can see there the image. Pleeeeeease, tell my something, that is insane! Thank you again!! By the way, im using Debian GNU/Linux 9.12 Client Version: v1.18.1 Server Version: v1.14.1

[@justmeandopensource]

HI Diego, thanks for watching. The issue you need to focus is around these lines "Failed to pull image "localhost:5000/myalpine:v1": rpc error: code = Unknown desc = Error response from daemon: Get localhost:5000/v2/: dial tcp [::1]:5000: connect: connection refused" The k8s node is having issues connecting to port 5000 on localhost which suggests that the registry service is not actually running. Let me ask you few things. 1. Where is your docker registry running? What is the IP address of this server? Is this your host machine? 2. How did you set up your k8s cluster? Are k8s nodes, virtual machines? Try changing the "localhost:5000/myalpine:v1" in your mainfest to "ipaddress:5000/myalpine:v1" with ip address of the machine where private docker registry is running. Cheers.

[@esse7847]

Hey Venkat, i'm getting this error "Failed to pull image "ip_address:5000/valiyev/myalpine:v1": rpc error: code = Unknown desc = failed to pull and unpack image "ip_address:5000/valiyev/myalpine:v1": failed to resolve reference "ip_address:5000/valiyev/myalpine:v1": failed to do request: Head ip_address:5000/v2/valiyev/myalpine/manifests/v1: x509: cannot validate certificate for ip_address because it doesn't contain any IP SANs". I'm using Docker Nodes (kuberntes Cluster created with kind). You have any idea how to fix it ?

[@justmeandopensource]

Hi Ali, Thanks for watching. Since you are running private Docker registry, you may not have set up ssl/tls on your docker registry or you might have used self-signed certificates. In either case, any docker client trying to connect to that private Docker registry will fail due to certificate errors. You will have to add insecure-registries options with the ip address of the machine where your private registry is running and then restart docker service. Check the solution here. github.com/kubernetes-sigs/kind/issues/110

[@StillMusician]

Bless you (couldn't withstand it).

[@justmeandopensource]

Hi Thomas, thanks for watching.

[@harshgoyal6822]

2 persons who disliked this video, their accounts should be deleted from youtube

[@justmeandopensource]

Haha. I love this comment.

[@soumyakantikar1152]

When using kubectl to create a pod and deploy the image it gives the error: Failed to pull image "192.168.100.11:5000/myorg/image:1": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.100.11:5000/myorg/image:1": failed to resolve reference "192.168.100.11:5000/myorg/image:1": failed to do request: Head "192.168.100.11:5000/v2/myorg/image/manifests/1": http: server gave HTTP response to HTTPS client. But if I perform docker pull from the worker node, then it pulls the image from the registry in the master node vm. Any idea as to what is the issue in here?

[@niketjoshi9234]

Hi I have tried these steps to create local registry but i am getting the below error. rpc error: code = Unknown desc = failed to pull and unpack image "MYIP:5000/sampleflask:1.0": failed to resolveMYIP:rence "MYIP:5000/sampleflask:1.0": failed to do request: Head "MYIP:5000/v2/sampleflask/manifests/1.0": x509: cannot validate certificate for MYIP because it doesn't contain any IP SANs Could please tell my what could be the reason