[ Kube 30.1 ] Kubernetes 1.17 on LXC Containers

[@tsingh7491]

Do you have kubernetes installed on the HOST ubuntu machine as well? I assumed this was not required. @10:33 I don't have kubectl or kubernetes installed on HOST.

[@justmeandopensource]

Hi Singh, thanks for watching. You don't need kubernetes cluster on your host machine. I don't have one. All you need is the kubectl binary. Then you can use the kubeconfig file from respective cluster to connect to it.

[@tsingh7491]

@@justmeandopensource Thank you Venkat for this initiative. Really helps people who are new to kubernetes world. The fact this is being pushed hard by Google, and now available on AWS, Cisco Container platform & VMware makes it that much more important to skill up on. Love your video tutorials and keep them coming, Cheers!

[@justmeandopensource]

@@tsingh7491 Thanks for your interest.

[@anthonyparagas7890]

Great tutorial!... Now that I have the k8s cluster running on lxc containers.. how do i get new lxc containers into the k8s cluster and is it possible?

[@thannasip8001]

hi,when i use lxc got same product_uuid same as host but for k8s it should be different right? and its giving error => connection shutting down due to error: local "d2:80:d2:7a:f3:bf(kmaster)" and remote "d2:80:d2:7a:f3:bf(kworker1)" peer names collision ref: github.com/lxc/lxd/issues/4737

[@thannasip8001]

Hi any help?

[@justmeandopensource]

@@thannasip8001 I have been using this lxc environment for my k8s cluster for a very long time without any issues. Can you explain or paste the commands/outputs of all the steps you followed in pastebin.com and share it. I should first try to reproduce what you are seeing before I can start to dig further. Cheers.

[@thannasip8001]

@@justmeandopensource thanks bro,this was because of peer name collision when we use weave net as network plugin,they used prodcut_uuid and machine_id for peer name creation,since we can't change product_uuid, have changed and its working fine bro,thank you :-)

[@justmeandopensource]

@@thannasip8001 Cool. Good to learn this as I have never encountered it. Thanks.

[@rsrini7]

பொங்கல் வாழ்த்துக்கள் venkat ..

[@justmeandopensource]

மிக்க நன்றி. Wish you the same. Cheers.

[@makrand1584]

Hello Venkat, I am new to K8. 2 days back I was able to get cluster running having master (kmaster) and worker (kworker) node. I am now trying to add 2nd worker node (kworker2) on into that cluster. I executed script. It didn't add kworker 2 into cluster. Worker is not running any docker container at all. And don't get any flannel or cni0 IP on same. Its kubelet service is failing to run. journulctl -u kubelet shows - Feb 23 10:09:49 kworker2 systemd[1]: kubelet.service holdoff time over, scheduling restart. Feb 23 10:09:49 kworker2 systemd[1]: Stopped kubelet: The Kubernetes Node Agent. Feb 23 10:09:49 kworker2 systemd[1]: Started kubelet: The Kubernetes Node Agent. Feb 23 10:09:49 kworker2 kubelet[6136]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See kubern Feb 23 10:09:49 kworker2 kubelet[6136]: F0223 10:09:49.334804 6136 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file Feb 23 10:09:49 kworker2 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a Feb 23 10:09:49 kworker2 systemd[1]: Unit kubelet.service entered failed state. Feb 23 10:09:49 kworker2 systemd[1]: kubelet.service failed. On this worker - the /var/lib/kubelet doesn't exist. Any idea how to tackle this? I even tried creating new cluster. Futile. The kubelet just works fine on master. Its just that its failing on the worker nodes.

[@justmeandopensource]

Hi Makrand, thanks for watching. There is no magic in adding additional nodes. The bootstrap script tells you all you need to do. You must have missed something. You are using LXC containers right?

[@manikandans8808]

That's cool. I always love LXC. I'll be working on this and ask some questions regarding the Insufficient CPU. I got lots of doubts. Catch u ltr...Cheers. It's amazing to work with LXC.

[@justmeandopensource]

Thanks for watching. Cheers.

[@JamesHowe]

I have one question about how the K8s environment gets configured using your script. In your 'Kubernetes the Hard Way' there is a step where you generate SSL certificates which are needed by the K8s environment. In your 'easy way', there isn't anything in your script which does this. Since I know SSL certificates must be generated, where is that happening when using your bootstrap-kube.sh script? Thanks again for your great set of videos. I've learned a great deal in a short amount of time.

[@justmeandopensource]

Hi James, thanks for watching. In "The hard way", we learn every step in the process of creating and running a cluster. We set up every component and is a good starting point to understand the cluster internal working and how different components are inter-related. In my vagrant/lxc clusters, I used kubeadm init command to initialize the cluster which auto generates the certificates for us.

[@Xus26]

Hello Venkat, I've run all the steps and everything was fine, but at the end, once I've tried to deploy the nginx example I've got the Error:ImagePull and the pod never hit the status Running. i.imgur.com/kGux145.png The Error response from daemon says toomanyrequests. But it's impossible that i've reached the limit since I've just installed everything. Any ideas of why I can't pull images? I've also done that docs.docker.com/docker-hub/download-rate-limit/

[@Sannamanrao]

First of all great videos. Appreciate your efforts. Just facing an issue. I have ubuntu VM on windows 10. K8s cluster worked fine when I did set up. But after restarting windows machine, when I do lxc list I get running containers kmaster and kworker1 but, I do not see flannel network getting recreated. So when i do kubectl get nodes i get error "The connection to the server 10.223.182.89:6443 was refused - did you specify the right host or port?". Any idea why? should I only use vagrant to create create ubuntu machine?

[@Sannamanrao]

Same thing happening even if i use vagrant.... Solution: I had to recreate reinitialize K8s cluster, redeploy flannel network.

[@justmeandopensource]

Hi Sandeep, thanks for watching. It never worked for me. All my clusters are just for local dev purpose and I always spin up new cluster when I want it. With LXC, I tried snapshotting the current state of the containers and restored it which worked but not consistently.

[@thomasbecker9777]

Try execute on each lxc container: echo 'L /dev/kmsg - - - - /dev/console' > /etc/tmpfiles.d/kmsg.conf

[@antuelle78]

Nice work man, thanx for sharing :)

[@justmeandopensource]

Hi Michael, thanks for watching.

[@dhananjaypatangay4029]

Hi Venkat, I am running my Linux host on Virtualbox. How can I see the dashboard from my host? (host->VM->lxc)?

[@justmeandopensource]

Hmm. Thats a bit involved to achieve. You will have to set up port forwarding between your host and virtualbox guest.

[@dhananjaypatangay4029]

Also after restart of the Linux machine the LXC container should be back?

[@dhananjaypatangay4029]

Hi Venkat, Do the docker containers for Kubernetes cluster come up automatically after the lxc container is rebooted? I rebooted my Linux host which rebooted the lxc containers but the docker containers within lxc container did not come up. Anything that I am missing?

[@justmeandopensource]

@@dhananjaypatangay4029 I always had problem running k8s cluster when shutting down and restarting the host machine. I don't have any production systems. This is just me test environment. I don't keep it running. Just spin up a new cluster when I need it. I think it won't auto start. You will have first start the master node and then the worker nodes assuming kubelet service is set for auto start (systemctl enable kubelet)

[@dhananjaypatangay4029]

Thanks Venkat. What are the steps to bring up the containers manually? I restarted kubelet service manually but still cant see any containers running

[@AnilShrish]

I am on ubuntu16.04, tried using your lxc profile which gives me below error: $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.6 LTS Release: 16.04 Codename: xenial $ lxc --version 2.0.11 $ lxc launch images:centos/7 kworker --profile k8s Creating kworker error: Failed container creation: - images.linuxcontainers.org: Failed to load raw.lxc

[@justmeandopensource]

Hi Anil, thanks for watching. Have you got lxd installed? That is the REST API layer to interact with lxc containers. I have done couple videos on LXC/LXD which you might find useful. I have other viewers following this video without any issues.

[@AnilShrish]

@@justmeandopensource I did watch your other videos on lxc and had installed the specific lxc version using snap.

[@justmeandopensource]

@@AnilShrish Ahh okay, thought you didn't watch those. Okay, in that case, it looks like a problem with the profile. One of the profile setting is not working with your version of lxc. Can you launch the container with default profile (without specifying the --profile option)?

[@AnilShrish]

@@justmeandopensource I had to reinstall my lxc setup, this time I had success with lxc 3.18 one issue is there but after that script execution on the kmaster I get this issue: [root@kmaster ~]# kubectl get nodes The connection to the server 10.191.61.219:6443 was refused - did you specify the right host or port?

[@justmeandopensource]

@@AnilShrish I was using 3.18 version as well. Even last night I used it without any issues. You will have to specify the kubeconfig file. The kubeconfig file on the kmaster node is at /etc/kubernetes/admin.conf. Try this command. $ kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes You can also copy this kubeconfig file to your host machine as $HOME/.kube/config and then you can just run kubectl commands. .kube directory under your home directory is the default location it will look for kubeconfig file named "config" Cheers.

[@knightrider6478]

Good day Venkat, Can you make a video detailing on how we can correctly deploy the kubeapps helm chart made by BItnami? I have tried a couple of times following their documentation and from another sources from internet and i had no success in order to deploy the kubeapps which overall is a good app and very usefull. And as always you make a great job my making all this video series. Thank you and regards !

[@justmeandopensource]

Hi Knight Rider, thanks for watching. Kubeapps looks interesting. I will definitely give it a try soon and make a video. Shouldn't be difficult. Thanks for suggesting this topic.

[@justmeandopensource]

Hi Knight Rider, I gave it a try last night. I couldn't get it working. Documentation not sufficient. I have my kubernetes cluster and helm 3. Their documentation suggests to go with Helm 3, but deploying it as per their documentation tries to deploy tiller kind of resource which is definitely not in Helm 3. They recommend using their Bitnami Kubernetes sandbox cluster in the cloud where it might work. Will continue digging this further and see if I go any further. Cheers.

[@knightrider6478]

@@justmeandopensource Yeah, their documentation is not so precise and not cover all the scenarios. I have tried with both version of helm 2 and 3 and I had no success. I thought at certain point that there is something which I did wrong or not understand it correctly. Thank you for reply

[@justmeandopensource]

@@knightrider6478 No worries.

[@Naralux]

Hey Venkat, Thank you for these videos, they are incredibly helpful! I'm new to the world of K8S and was wondering if you have a video explaining why you use LXC containers versus something like microk8s or minikube? If not, could you tell me? Thanks!

[@justmeandopensource]

HI Ash, Thanks for watching. Basically I prefer LXC containers over Virtual Machines (using VIrtualBox or KVM) as they are light weight. I haven't done any videos specifically explaining why I prefere lxc over minikube or microk8s. I have Kubernetes Provisioning Playlist: kzbin.info/aero/PL34sAs7_26wODP4j6owN-36Vg-KbACgkT and Linux Containers Playlist: kzbin.info/aero/PL34sAs7_26wPfLNZ5NKoH0RKbTAWMlh2I Which could give you some insights. LXC containers are easy on the system using less resources with no overhead of hypervisor. I can run more lxc containers than virtual machines. Basically these are the reasons. Cheers.

[@Naralux]

@@justmeandopensource Ah makes sense, thanks! I'm currently binge-watching your Kubernetes playlist, it's an absolute goldmine!

[@justmeandopensource]

@@Naralux You are welcome.

[@Naralux]

@@justmeandopensource Another question: does your cluster resume normal operations after rebooting the host machine or do you have to provision after every host reboot? My cluster ceases to function upon host reboot. lxc list shows no flannel overlay and kubectl cluster-info gives a connection refused error.

[@justmeandopensource]

@@Naralux I don't manage any production grade clusters. Just for learning and development purpose. So I just spin up new cluster each time I need it. It sometimes fails to start when the host machine is rebooted but worked few times. So no consistency. You can use lxc snapshots feature to snapshot the current state of all the nodes and then restore it when you restart your host machine.

[@naturboytv6669]

Nice Video, I am still having trouble to install kubernetes inside lxc on my proxmox host. Maybe you can create a tutorial for that specific case? Proxmox has so nice features like easy backups and so on and I am not able to suspend the host atm. Still thanks for sharing your solution.

[@justmeandopensource]

Hi, thanks for watching. I am not aware of Proxmox and never used it. What I would suggest is to first try it on your local workstation if you have. When you say you have trouble installing kubernetes, what exactly is the issue? Could you please provide me more details. Thanks.

[@naturboytv6669]

@@justmeandopensource The main problem is that the kubelet doesn't come up during the initialisation phase. Unsure why exactly

[@justmeandopensource]

@@naturboytv6669 Checking the kubelet logs may give you more information.

[@jono_in_AU]

Hi Venkat, Greetings from New Zealand. Great video, one thing, when I ran the script to join the worker to the master it complained that it could not resolve kmaster.lxd from kworker.lxd, I just added the ip address for kmaster.lxd to /etc/hosts file on kmaster. Cheers Jono

[@justmeandopensource]

Hi Jon, thanks for watching. Hmm I never had that problem. Any lxc container can resolve any other lxc container using the format ".lxd". This is internal dns for lxd layer. May be we are using different lxd versions. Did you got it working after adding the ip address to the /etc/hosts file?

[@jono_in_AU]

@@justmeandopensource yep, worked like a charm. One other thing I had to also do is put that workaround for the /dev/kmsg in /etc/rc.local so on the container reboots they get created again. Just one other thing, if I wanted to use ubuntu containers instead of centos7, will just need to adapt the bootstrap script to install in ubuntu?

[@justmeandopensource]

@@jono_in_AU Cool. Yes the bootstrap and provisioning scripts were designed for CentOS 7, but you can modify the commands to suit Ubuntu. But I don't care about the underlying instances. They are just worker nodes and any stable linux distro should be fine. Cheers.

[@karlovasiradio]

[TASK 1] Install containerd runtime bash: line 10: containerd: command not found Failed to restart containerd.service: Unit not found. [TASK 2] Add apt repo for kubernetes (23) Failed writing body [TASK 3] Install Kubernetes components (kubeadm, kubelet and kubectl) [TASK 4] Enable ssh password authentication Failed to restart kubelet.service: Unit not found. sed: can't read /etc/ssh/sshd_config: No such file or directory Failed to reload sshd.service: Unit not found. [TASK 5] Set root password [TASK 6] Install additional packages [TASK 7] Pull required containers [TASK 8] Initialize Kubernetes Cluster [TASK 9] Copy kube admin config to root user .kube directory cp: cannot stat '/etc/kubernetes/admin.conf': No such file or directory [TASK 10] Deploy Flannel network [TASK 11] Generate and save cluster join command to /joincluster.sh I took some error when i execute the command cat bootstrap-kube.sh | lxc exec kmaster bash Do you know why? I am also using centos7 image.

[@justmeandopensource]

Hi, did you check the recent video I did on this? kzbin.info/www/bejne/pJezl2Omf5aMgqs

[@karlovasiradio]

Same for ubuntu20.04 images [TASK 1] Install containerd runtime [TASK 2] Add apt repo for kubernetes bash: line 15: curl: command not found [TASK 3] Install Kubernetes components (kubeadm, kubelet and kubectl) Failed to restart kubelet.service: Unit kubelet.service not found. [TASK 4] Enable ssh password authentication sed: can't read /etc/ssh/sshd_config: No such file or directory Failed to reload sshd.service: Unit sshd.service not found. [TASK 5] Set root password [TASK 6] Install additional packages [TASK 7] Pull required containers [TASK 8] Initialize Kubernetes Cluster [TASK 9] Copy kube admin config to root user .kube directory cp: cannot stat '/etc/kubernetes/admin.conf': No such file or directory [TASK 10] Deploy Flannel network [TASK 11] Generate and save cluster join command to /joincluster.sh

[@karlovasiradio]

@@justmeandopensource you saved my day! I will check that!

[@justmeandopensource]

@@karlovasiradio No worries. I only recorded this video recently and it is a working setup which I verified.