Understanding Kubernetes Networking. Part 2: POD Network, CNI, and Flannel CNI Plug-in.

In this video, we will build on what we learned from the container networking presentation and will discuss an overview of the Kubernetes networking model and POD networking details. We will then explore Container Network Interface (CNI) and how it works. We will finish the presentation with a deep-dive exploration of the Flannel CNI plug-in.
Complete playlist for these series: • Kubernetes Networking ...
Keywords: Kubernetes networking model, POD networking, Flannel, CNI, POD Network, Overlay Networks, Kubernetes CNI
My Other Videos:
► Cilium Kubernetes CNI Provider, Part 1: Overview of eBPF and Cilium and the Installation Process • Cilium Kubernetes CNI ...
►Cilium Kubernetes CNI Provider, Part 2: Security Policies and Observability Leveraging Hubble
• Cilium Kubernetes CNI ...
► Cilium Kubernetes CNI Provider, Part 3: Cluster Mesh
• Cilium Kubernetes CNI ...
► What is VXLAN and How It is Used as an Overlay Network in Kubernetes?
• What is VXLAN and How ...
► Managing Linux Log-ins, Users, and Machines in Active Directory (AD): Part 2- Join Linux Machines to AD:
• Managing Linux Logins,...
► Managing Linux Log-ins, Users, and Machines in Active Directory (AD): Part 1- Setup AD:
• Managing Linux Logins,...
► Sharing Resources between Windows and Linux:
• Sharing Resources betw...
► Kubernetes kube-proxy Modes: iptables and ipvs, Deep Dive:
• Kubernetes kube-proxy ...
►Kubernetes: Configuration as Data: Environment Variables, ConfigMaps, and Secrets:
• Kubernetes: Configurat...
►Configuring and Managing Storage in Kubernetes:
• Configuring and Managi...
► Istio Service Mesh - Securing Kubernetes Workloads:
• Istio Service Mesh - S...
► Istio Service Mesh - Intro
• Istio Service Mesh (si...
► Understanding Kubernetes Networking. Part 6: Calico Network Policies:
• Understanding Kubernet...
► Understanding Kubernetes Networking. Part 5: Intro to Kubernetes Network Policies:
• Understanding Kubernet...
► Understanding Kubernetes Networking. Part 4: Kubernetes Services:
• Kubernetes services - ...
► Understanding Kubernetes Networking Part 3: Calico Kubernetes CNI Provider in-depth:
• Understanding Kubernet...
►Understanding Kubernetes Networking. Part 1: Container Networking: • Video
► A Docker and Kubernetes tutorial for beginners:
• A Docker and Kubernete...
► Setup a "Docker-less" Multi-node Kubernetes Cluster On Ubuntu Server:
• Setup a "Docker-less" ...
►Step by Step Instructions on Setting up Multi-Node Kubernetes Cluster on CentOS: • Step by Step Instructi...
►Setup and Configure CentOS Linux Server on A Windows 10 Hypervisor - KZbin: • Setup and Configure Ce...
►Setup NAT (Network Address Translation) on Hyper-V: • Setup NAT (Network Add...
► Enable Nested Virtualization on Windows to run WSL 2 (Linux) and Hyper-V on a VM: • Enable Nested Virtuali...
►Setup a Multi-Node MicroK8S Cluster on Windows 10: • Setup a Multi Node Mic...
► Detailed Windows Terminal, (WSL 2), Linux, Docker, and Kubernetes Install Guide on Windows 10:
• Detailed Windows Termi...

[@danielqian8899]

This is one of the best in-depth demo of how kubernetes networking works. Other youtubers talk in a high level way that I can understand perfectly each every sentence they say but don't know exactly where it fits in my knowledge accumulated over nearly two decades of networking and system working experience I was wondering how two different subnets talk to each over VxLAN and from your video it looks like they do by source natting the original pod IP to that of the flannel tunnel IP

[@TheLearningChannel-Tech]

Hi Daniel, many thanks for your kind words! Correct , Flannel performs SNAT of the originating POD. Thanks again for your feedback!

[@danielqian8899]

@@TheLearningChannel-Tech Thanks for the response that is very helpful. I still don't understand the reason why the source needs to be translated in both Flannel and Calico cases as shown in your demonstrations. From what I can see, in both forward and revere directions the routes to the pods on the other node are symmetrically available so technically communication can happen without nat translations.

[@TheLearningChannel-Tech]

Hi @daniel qian. In both cases (Calico and Flannel) the packet capturing shows that they perform SNAT. You can see that in the Flannel video (kzbin.info/www/bejne/i2SYdGN7hbiqpbs). I haven't had time to research why that's the case, I'll update this post if I find the answer. In Calico case I was running it in "ipip" mode and the Calico doc says the following: "Note: Setting natOutgoing is recommended on any IP Pool with ipip enabled. When ipip is enabled without natOutgoing routing between Workloads and Hosts running Calico is asymmetric and may cause traffic to be filtered due to RPF checks failing." (docs.projectcalico.org/reference/resources/ippool). " Hope this helps. Thanks again for your questions and comments!

[@danielqian8899]

@@TheLearningChannel-Tech Thanks for the great information again. I will do some research as well and update if I ever find anything.

[@AnkushSharma-xj2ng]

@@TheLearningChannel-Tech As you explained and as outlined in Kubernetes networking rules, NAT shouldn't be involved in pod-to-pod communication. However, you mentioned that SNAT is occurring here-can you clarify why this is happening? Thank you for posting such an amazing video and creating great content on Kubernetes networking.

[@TheHolycrack]

This is the kind of content I want to see every time I'm doing a deep dive into something and want to improve my understanding! Thank you!

[@TheLearningChannel-Tech]

Glad it was helpful!

[@atulmahori2497]

I must say that this playlist is the best on Kubernetes networking, starting with the lowest layer till the highest layer.

[@touseefyousuf3232]

Hats off to you for such an insightful content delivered in this k8s networking series, stating this from the background that I have used k8s for at least 5years now. I will already make you CTO of my company if I had one😊

[@rakibhossain2536]

best explanation. Thanks for making such kind of contents

[@rafazmyslony5819]

Really, it's the best kubernetes networking explanation on entire internet. Believe me, I've seen many, even in diffrent languages:D

[@TheLearningChannel-Tech]

Glad it was helpful!

[@carlossanchez4798]

Thank you so much so doing these series! This is the best K8s networking explanation I have found while trying to learn about this topic!

[@TheLearningChannel-Tech]

Glad you like them!

[@adeshkashyap1703]

Being K8S Admin guy .. here I found the best on K8S networking .. Liking your videos too much .. Thanks for this great work

[@AnujKumar-yy1gk]

Excellent video. Great explanation. This is the video I was looking for.

[@rebelmoon-aj]

Of all the videos on K8 in youtube, this one stands out.

[@TheLearningChannel-Tech]

Thank you, Ajit, much appreciated!

[@jaxx4fun]

Love this tutorial very much for it's detailed enough so that I could focus on the video without switching out to search additional information. Thank u very much. With your talk, I feel like the networking is not mystery any longer and everything under the cover has its footprint.

[@jaxx4fun]

One question is that the source IP masquerading as the tun ip FMHO looks like NAT, if so, doesn't it break the rule NO NAT mentioned in the part 1?

[@TheLearningChannel-Tech]

@@jaxx4fun Hi, no NAT means that as far as PODs are concerned, they can connect directly to other PODs without having to worry about natting. The overlay network does that seamlessly and PODs are oblivious to that.

[@rewantasubba5180]

man this is an amazing tutorial. As a networking pro, l liked every bit of presentation. Thanks for sharing this awesome lesson.

[@TheLearningChannel-Tech]

Glad it was helpful!

[@ahmedsaid-s2w]

Super, this is by far the best explaination, keep it up.

[@tracylee8446]

Hi There, it is me again, I just gave up that issue in previous video, and move forward here to keep my learning. I believe there is no big problem so thank you .

[@TheLearningChannel-Tech]

Hi Tracy, thanks for the update. Sorry, the issue wasn't resolved but that shouldn't prevent you from moving forward. Raw container communication is a bit involved but fortunately, CNIs (Container Network Interfaces) hide a lot of that complexity in Kubernetes. We can revisit the issue in the future if needed, I'll just need a bit of info such as the command that caused the issue. Thanks and take care, Gary.

[@rsrini7]

No where seen this kind of detailed explanations with diagrams. Bow to you. Thanks a lot.

[@TheLearningChannel-Tech]

Glad it helped

[@jassspecial7535]

This is really impressive. Pretty much explained in details. I'll now watch more videos! keep posting more like this!

[@TheLearningChannel-Tech]

Thanks for your feedback!

[@stephenplautz4682]

This video has been super helpful for configuring my cni on my personal cluster project, and also just for understanding networking more in general. Great job!

[@TheLearningChannel-Tech]

Hi Stephen, thank you for the kind words and glad you fount it useful. Thanks again!

[@andydufresne9387]

Your tutorial was exemplary. 😄Keep up the exceptional work!

[@TheLearningChannel-Tech]

Glad you liked it!

[@adityask22]

The content is pure gold.

[@TheLearningChannel-Tech]

Thank you!

[@sumanreddyk7284]

Brilliant!! Excellent deep dive explanation. Please share your knowledge and experience with us. Thank You! very much.

[@TheLearningChannel-Tech]

Thank you!

[@carlche647]

This is really grate demo of k8s network in-depth.

[@santosharakere]

Amazing demo and details. Thank you very much.

[@xcage90]

Thanks for the tshark demonstration, well explained

[@TheLearningChannel-Tech]

Glad it was helpful!

[@soahibasif659]

Very detailed and nice explanation.

[@vikieethechip]

Wonderful Explanation. One of the best i would say.

[@TheLearningChannel-Tech]

Glad it was helpful!

[@mrwhitebp]

omg man this is soooo good video. I definitely going through all of them congrats

[@TheLearningChannel-Tech]

Thank you very much, glad it was helpful!

[@sandeeprazdan1976]

Excellente thumps up for your hard work

[@TheLearningChannel-Tech]

Thank you! Cheers!

[@mahi2082]

No words just Awesome i can say.....

[@TheLearningChannel-Tech]

Thank you very much for kind words!

[@grjesus9979]

When you show interfaces attached to the bridge where is 10.244.1.1?? As in the diagram it should be mentioned, yet it is not. I think it is because bridge creates a local network for pods within the node. To access the internet and pods in other nodes kubernetes services are required.

[@aylacaliskan5596]

Your tutorials are awesome..Thanks alot

[@TheLearningChannel-Tech]

Thank you very much, glad you like them!

[@jeetak47]

Very rare content!

[@manjitkaur8877]

great information , Thanks for sharing !!

[@TheLearningChannel-Tech]

Glad it was helpful!

[@avinashsabat]

amazing video, such clear explanation!!

[@TheLearningChannel-Tech]

Glad it was helpful!

[@mohamedhanif6035]

Great work. Thank you

[@sandeeprazdan1976]

Very good keep it up

[@eldarkarimov5791]

hi thanks for this helpful video I have a question : How does work default route in pod and which interface this ip dedicated set up ? where send all packets this default gateway after receive packets from pod that connect throughout veth interface to the bridge ?? Could you answer my question please ?

[@vitorguidie]

Whoever you are, I love you lol

[@nikhilsrivastava9120]

can we dynamically update VNI of the egressing packet depending on which veth pushed the packet into the bridge?

[@bhaaratsharma6023]

This is the most amazing pod networking tutorial I have ever seen. I have been following this playlist and your other tutorials related to OSI helped me understand basics of networking really well. Thank you very much. I have one question. In this if the Flanel 1 network is present, why do tshark logs show data flow from eth0 or node 1 to eth0 of node 2? Doesn't the data flow directly through flannel tunnel? Am i missing something or my understand of an overlay network is wrong?

[@TheLearningChannel-Tech]

Hi, thanks for your feedback! So, you are saying Flannel 1 does not SNAT the calling POD IP? If all your worker nodes are on the same subnet, then the newer Flannel CNI might be smart enough to bypass the tunnel and take advantage of Linux's direct routing. The video was recorded last year, it is possible that the Flannel plug-in might have gone through changes. when I tested this, with all my worker nodes being on the same subnet, it still went through the tunnel as you saw in the video. Overlay networks do add to the overhead so if they can be avoided, that would be a plus. Hope this helps.

[@spiraldynamics6008]

Can you provide me link for OSI tutorial please?

[@AnuragSharma-kb9pc]

Thank you so much for such good content

[@TheLearningChannel-Tech]

Thank you very much for the kind words and glad you found it useful!

[@indranilbanerjee5521]

Wonderful explanation Sir

[@TheLearningChannel-Tech]

Thank you!

[@jayeshthamke7860]

Great job!!

[@TheLearningChannel-Tech]

Thank you! Cheers!

[@mohdnahidmondol]

Wonderful Explanation. how to know which veth attached with which pod eth0. For example kube-node1-fl pod eth0: 10.244.1.8 attached with veth23a4a1a2

[@TheLearningChannel-Tech]

Hi, There is no direct way that I know of to tie the two halves of the veth set together. Deploy a POD with replicas set to 1 and take note of the name of veth half on the node by running: ip link show type veth If you are experimenting with more than one number of replicas for that POD then set the number of replicas to two and record the second veth, and so on....

[@vy5644]

Thanks a lot this was really helpful. Why does the packet going from Worker node pod to Master node pod have a destination mac address of the Flannel1.1 interface instead of Master node pod interface mac address? I thought Vxlan is a layer 2 tunneling technology.

[@TheLearningChannel-Tech]

@VY, hello and thank you! There are two ethernet frames involved here, the outer ethernet frame embeds the POD to POD communication. The outer frame is sent through the tunnel, when it arrives on the other end, the other server is not aware that the received frame has another frame embedded within it, I think this is where the Flannel1 interface intercepts the message and delivers the last leg of the journey which is the POD that initiated the call to other side. Hope this helps. Thanks again!

[@vy5644]

Alright. I will set up the environment to play around with it. Thanks again for the detailed explanation this is by far the best one I have seen.

[@arkasharma]

Thanks for this brilliant explanation. I have a small question. In the beginning you mentioned about having no NAT, but around 32:38 when you dumped the captured frames we have the source pod IP changed to the Flannel 1 IP. Isn't that some kind of NAT ? Thanks

[@TheLearningChannel-Tech]

Hi, yes that is correct.

[@davidlira8853]

Amazing, I really appreciate the diagrams

[@TheLearningChannel-Tech]

Many thanks for your kind feedback!

[@453nabeel]

Hello Sir , after installing Kubernetes Cluster using 22.04 Ubuntu LTS , we have 1 x master and 2 x worker nodes. We see calico as CNI plugin installed as a part of installation. Is it possible to change it to flannel instead of Calico and see the things that you did. It was great see VXLAN VNI in packet capture. Thanks !

[@TheLearningChannel-Tech]

Hi, which episode are you referring to? All CNIs provide the same basic functionality but differ in implementation. So, if you just want to set up a cluster, any CNI would work.

[@453nabeel]

@@TheLearningChannel-Tech Thank you for replying. I am referring to Episode 2 of the series where you demonstrate how Flannel uses VXLAN for POD to POD communication across nodes. When I install cluster I see Calico as CNI Plugin. In order to see flannel , can I change to Flannel and then revert back to Calico?

[@TheLearningChannel-Tech]

@@453nabeel You'll have to uninstall Calico and then install Flannel. There is no way to swap CNIs.

[@453nabeel]

@@TheLearningChannel-Tech is there any guide that shows how to remove Calico and install flannel? Because after testing flannel I will revert back to Calico by again installing it

[@zeesiddiqui6473]

Great Video!! Are Flannel1 and Flannel2 IPS are in the same subnet at 12:55? What are VTEP IPS? Flannel IPS or Node eth0 IPS? Thanks!!

[@TheLearningChannel-Tech]

Hi, No, Flannel1 and two in two different subnets as shown on the screen (one is in 192.168.0.x and the other in 192.168.1.x). Flannel1 and 2 are the VTEPs.

[@mubashir1976]

@@TheLearningChannel-Tech I saw flann;el1 and flannel2 MAC address inside UDP as payload in your capture. How does Flannel1 finds the MAC address of Flanel2 as ARP can not be used because they are being in different subnets? Much appreciated!!

[@mrwhitebp]

Questions for you , Is this always the same port 8472, what is the services that supposed to be listening on that UDP port? and you show me a netstat -antup and the socket listening ? I have two servers that I think, they are not establishing the tunnel flannel.1 inet 10.244.59.0/32 and cni0 inet 10.244.59.1/24

[@TheLearningChannel-Tech]

Hi, Flannel listens on UDP port 8472 for VXLAN traffic. The official assigned port for VXLAN is "4789" but Flannel for some reason uses port "8472". Please make sure that UDO port "8472" is open if you have enabled the firewall and also make sure both VMs can ping each other as VXLAN relies on L3 connectivity. #To view which UDP ports are open, please run: netstat -lntu #To monitor VXLAN traffic you can install "tshark" and run the following command, watch the video on how to run it and waht to look for: sudo tshark -V --color -i eth0 -d udp.port=8472,vxlan -f "port 8472" Good luck.

[@yalappahero5912]

How does the frontend talks with backend in k8s as backend is cluster ip and frontend how does it connects with backend

[@TheLearningChannel-Tech]

I'm not quite sure what you are asking. Simply put, a Kubernetes service is simply a load balancer. When the client and servers are on the same Kubernetes cluster when a client calls the ClusterIP of the service, one of the PODs behind the service is randomly selected by Kubernetes and answers the call. Makes sense? If you want to better understand Kubernetes services, watch this: kzbin.info/www/bejne/eIvOY3uLgNird7M Hope this helps.

[@yalappahero5912]

@@TheLearningChannel-Tech I will tell all my requirements as I want to deploy a three tier project that is mern stack I have connected backend with mongodb as my frontend is react and backend is nodejs I have to follow the previous reply that you have suggest I am Right

[@yalappahero5912]

Basically you are telling that backend as a nodeport and frontend as load balancer as service type

[@TheLearningChannel-Tech]

@@yalappahero5912 No, I think it would be best if you watched the Kubernetes service I linked in the previous post. ClusterIP is the load balancer as I explained earlier and is used by clients that are inside the cluster and want to communicate with another pod (service) inside the cluster. Clients outside the cluster can communicate with a service inside a cluster through NodePort. NodePort internally calls the services ClusterIP.

[@user-mb7qe6ro9m]

What if we don't have a CNI plugin, will pods still be able to communicate with each other hosted on different nodes via kubeproxy?

[@TheLearningChannel-Tech]

No, a CNI plug-in is required to set up the POD network. Kube-proxy's job is to set up a ClusterIP for service and program the iptables to route the call through the POD network (setup by CNI) to a POD. So in short, a CNI plug-in is needed for your cluster to function.

[@AvinashReddy21]

Excellent video ! Part 1 of the networking video is unavailable

[@TheLearningChannel-Tech]

@Avinash Reddy, thank you for feedback! I just tested Part 1 and it is working for me. Do you get any specific errors? Could you try again please? Thank you!

[@pengdu7751]

It would be probably helpful to explain what "ip route" is supposed to show at 19:37. It's executed on one node without being give any other node. what route is it showing? without this it's easy to get boggled down when you explain the route outputs.

[@TheLearningChannel-Tech]

@Peng Du, thank you for the feedback! The route I'm showing is for when a POD on one node talks to another POD on the same node, that is the communication goes through the bridge. Sorry if that caused confusion. I apologize if I misunderstood your post, if that's the case, please repost. Thank you!

[@vuhoanghiep1993]

Thank you for your video, this base from scrach so very useful for newbie like me, can you explain how to setup enviroment to run command in visual studio code like you, just some keyword for google, thanks so much

[@TheLearningChannel-Tech]

@vuhoanghiep1993, thank you for your kind words. You can follow this page to set up your VS code remote SSH: code.visualstudio.com/docs/remote/ssh-tutorial. Hope that helps.

[@TheLearningChannel-Tech]

Forgot to mention that the remote VS ssh link I provided is useful when you want to access the remote machine also, for instance if you have scripts on that machine. If that's not required, you can ssh to that machine as the first line of code in VS, for example "ssh gary@10.0.0.144". In both cases the ssh server must be running on the remote. Let me know if you further questions. Cheers!

[@ClarityAndHappiness]

Its gold

[@TheLearningChannel-Tech]

Thank you, glad you found it useful.