Wildcard Certificates with Traefik + cert-manager + Let's Encrypt in Kubernetes Tutorial

[@TechnoTim]

Are you using certificates yet for your services??

[@CRK1918]

I using treafik as I fallow you video(internal and external), this video is upgrade from that. Thanks

[@christiandassy8128]

Thank you very much for your videos I really loved them I have been seeing them for over 2 years now! So big fan....Always help me push my boundaries and to learn and have a better self-hosted setup. Quick question, are you still using longhorn?

[@cxl520]

I also use your old video guide and work great for me! I'm still relatively new to this video setup and I have some questions, where to put traefik dynamic config files (such as middleware and services IP)?

[@majorgear1021]

I'm getting there. From my other thread, I had Traefik in docker serving certs for 2 domains. I'm transitioning to Traefik in a k3s cluster, but there is a learning curve to get it working. it isnt' as easy as copying and pasting from config.yml into a manifest file.

[@RichDurso]

Suggestion for people wanting to have HA Traefik, if your K3s is already HA (3 or more master nodes) then consider setting Deployment "Kind" to DaemonSet. This will automatically place 1 and only 1 Traefik instance per master node making it actually HA (it creates the nodeSelector). Just bumping replicas to 3 (without setting affinity/anti-affinity) could put all 3 instances on one node which is not an HA configuration. Awesome video as always!

[@TechnoTim]

Thanks, will add affinity to the example!

[@TechnoTim]

Also, the downside of using DaemonSet is that it will run on n nodes, which might be more than you want. I use affinity in my env, otherwise I would have too many traefik pods running! Will update the example!

[@majorgear1021]

+1 on good point. Do you need multiple instance of Traefik for availability if you have Metallb? eg, if I disconnect the node currently running Traefik, won't controllers just start up new instance on another node? There might be a delay during that time it takes the new Traefik instance to start, but it would happen eventually, right? Or am I missing something.

[@geemobile6037]

@@majorgear1021 the point of the high availability is so you don’t have that down time between the single pod going down then back up. You can use it without HA if your services don’t require zero downtime. I currently run my self hosted services this way. Later on I’ll switch to HA.

[@packetgeek]

@@majorgear1021 If you're using a local registry to store images, you could push the image to it and minimize load time. Of course, once the image has been loaded onto a node, (unless you've configured K8S to delete unused images from the nodes' cache) it will remain in the node's cache. This mod requires some manipulation of the helm chart/manifests but might be worth the effort if you have a lot of node failures, Internet connectivity issues, or similar outages.

[@nabajaffry9321]

You look tired man. Really appreciate the hard work you put in for this video. The level of detail is really something you cant find anywhere else.

[@TechnoTim]

Thank you! TBH I was so tired, didn’t feel well but the show must go on! Thanks for noticing!!!!

[@TeChn4K]

Just a note to viewers and you Tim : `commonName` in Certificate object is deprecated. At least one dnsNames is enough. Thanks for this very detailed and comprehensive video

[@victorwinkler2727]

I have been trying to get this to work for a good 5 Days now. THANK YOU for colecting, presentingand explaining all this information. You are awesome.

[@TechnoTim]

Thank you!

[@dadrad]

Great job explaining the DNS01 challenge for self hosted DNS. I wish this video existed 4 months ago, lol!

[@TechnoTim]

Glad it was helpful!

[@BrianSez]

Hey Tim, great tutorial! Would you consider creating a video on how you backup your data? Or perhaps a sort of 'best practices' video on data backup?

[@llortaton2834]

I feel like you have been teasing this tutorial for YEARS, thank you Tim, a lot.

[@esaenz7]

Awesome! This came just in time after spending this week learning about certificates and how to apply them to my services. Still a lot to learn and practice... Thanks!

[@ran_red]

This is one of the areas I struggle with the most, cluster networking in general. You make it easy to understand so thanks for that

[@chrisa.1740]

This is just what I needed to move from my Docker + Traefik + Cloudflare setup that never really seemed to work. Thanks for the vid!

[@ukaszl.9943]

This is the best tutorial about kubernetes, that I ever seen!!! You great, better than ChatGPT :) Thank YOU very much, this is what I needed. Everything works like a charm. Great job. Than you, thank you, thank you. You are great :)

[@randleqgod]

I’m gonna come back to this when I outgrow Traefik with Docker. This looks amazing.

[@ch40sth30ry]

Freaking awesome Tim. Completely demystified the process for me and am currently using it in MY 'production' environment. Thank you!

[@gravyflex]

This was such an excellent video. You are really good at explaining things. I keep coming back to this video, I've seen it more than five times already. I've wanted to set this up for a long time and I am happy with the results.

[@rileydavidjesus]

I like that you call this 'homelab' this is enterprise grade production work.

[@primeix]

Oh Man is this the upgrade video to the Docker Version? But for Kubernetes?!?! I am soo excited It's like I been waiting for this video to move forward with my lab...

[@TechnoTim]

Yup! Nailed it!

[@vladimir5935]

I use your k3s-ansible playbook extensively and find it very useful. Would you consider adding a feature to allow adding nodes to the cluster via ansible?

[@Kessra]

Just a word of warning. If your password contains special characters like a $-sign you need to escape that sign with a leading backspace --> \$ within the htpasswd command: 'htpasswd -nb user pa\$\$word | openssl base64'. Further note: it doesn't change the outcome whether or not you put the password between quotation marks in the htpasswd command. While the string you should paste to the secret-dashboard.yaml file is a bit different, the actual basic-auth challenge will accept the password without quotation marks anyway

[@mitchross2852]

Ok I had to watch this video 5 times in a row. I totally get it now.

[@haventfoundme]

Excellent breakdown Tim. Much appreciated.

[@jonasdamfors8249]

Really love your videos. Especially the k8s/dev tools ones like grafana and it's companions

[@TechnoTim]

Thank you!

[@CrashLoopBackOff-K8s]

I think my previous comment was deleted, perhaps due to the label being interpreted as a link. In any case, quick repost: 1. Love your videos and all that you give back to the community -- thank you! 2. When you were tailing the logs for the cert-manager pods, you don't have to look at them individually. If you use the label for the controller pods, you can look at or tail them all simultaneously. Here's an example: kubectl -n cert-manager logs -l="put the pod labels here" -f. You can get the pod labels by doing a kubectl -n cert-manager get po --show-labels. I'm guessing you already know about this, but passing it along just in case. 3. When it comes to the helm commands, a couple of things I do to reuse the same commands so that they work whether I'm upgrading or installing for the first time: "helm upgrade --install --namespace=traefik --create-namespace traefik traefik/traefik --values=values.yaml" Using upgrade with the "--install" option allows you upgrade the release if it exists, or install it if it does not with the same command. Similarly, passing "--create-namespace" will create the namespace for the release if it does not exist, which can save you some time. These may or may not be useful, but passing them along just in case.

[@TechnoTim]

Thank you! Yes, automod blocks anything that looks like a link! Thanks for the helm upgrade/install command. It use it but should more often!

[@CTWilliams89]

Wanted to say thank you again for this video, my cluster has been chugging along but failed to renew the cert recently. I used this video to help remedy the issue! Have you thought about doing a video on upgrading a k3s cluster? I realized mine is now way out of date since spinning it up with your playbook a year ago lol!

[@packetgeek]

Took me about a day to get this working (can type about as well as I can dance) but it's online now. Doing a write up of my notes.

[@TechnoTim]

Nice work! 💃🕺

[@packetgeek]

@@TechnoTim Thanks! Grabbing your launchpad repo from Github helped. I'm slso working up notes on what needs to be done to add other services. I'm thinking that these should go through your same staging-first/production-second approach. Unfortunately, the constrained resource (as usual) is time so it's not at the top of the "to do" liist. Dex SSO deployment is at the top. Also need to walk a friend through deploying the traefik+cert-manager rig.

[@packetgeek]

Got it working nicely with Authentik now.

[@Faithtosin]

Amazing and a very helpful video. You're amazing Tim.

[@RichDurso]

Comment on the Traefik HA limit (16:25 in video). [My apologies, if you were keeping things simple. Just want to keep you honest]. The limit is not due to the storage or PVC as you hinted. That is old Traefik 1.x days and was removed in Traefik 2.x. The issue is because there is no way to insure the correct instance of Traefik will receive the Challenge Request and subsequent responses when you have more than one instance. It is just not possible to run multiple instances of Traefik with LetsEncrypt enabled. You need something else like Cert-Manager to handle the certs as you demonstrated.

[@TechnoTim]

Ah! Thank you! Also, I couldn’t mount the PVC to be read/write many also limiting me to one replica!

[@carlitros1207]

random question, if you want to add the certificate to the traefik dashboard, do you also need to make a certificate in the traefik namespace? or how does that work?

[@hayanradwan6100]

I'm wondering this as well

[@JonathanJensenp]

Great tutorial as always. This was very informative and helpful. Keep up the great work.

[@TechnoTim]

Thank you!

[@f1aziz]

Damn, this was not a walk in the park. Thanks.

[@sachasmart7139]

Amazing. I learned so much. Thank you for all you do.

[@fredrik354]

This is awesome Tim, thank you very much!

[@techchad9730]

Hey tim, your videos are awesome, it helped in many different ways, can you make a video on installing rancher using helm on k3s v1.24.3+k3s1, it feels like banging my head in the wall.

[@zavarka2]

Thanks, Tim. This helped me.

[@isLife-if8lz]

This is amazing! Thank you!

[@michaelhenry1857]

Thanks for the great video! My current setup is based on your previous video "Put Wildcard Certificates and SSL on EVERYTHING - Traefik Tutorial" and I am excited to move this functionality into the cluster. But I am stuck on one thing - how do you implement the Authelia and Crowdsec middleware with this method (I'm assuming they have to have their own deployments?)

[@ws_stelzi79]

It somehow sounds like SOMEONE has learned the hard way why you should use the staging thingy in Let's Encrypt. 😉😇

[@TechnoTim]

I test in prod!

[@ws_stelzi79]

@@TechnoTim Only the best do that!

[@cwvhogue]

Thanks for the great how-to video and notes! I have a k3s cluster that fetches daily certificates from a - homelab internal Smallstep "step" certificate authority on a raspberry pi. And an internal BIND9 DNS server for challenges with my non-public homelab domains. These work the same way, same protocols as Letsencrypt and Cloudflare DNS - but provide a way for my k3s setup to use internal-only domains with homelab certificates via a similar setup to yours. I have to put my own homelab public cert into my browsers, but only once as all my certs, while self-signed, come from a homelab root certificate.

[@TechnoTim]

Nice! I will have to look into smallstep!

[@UntouchedWagons]

Just got it working. I did have an issue with nginx still using the staging cert, but that's because I forgot to change the cert its ingress route was to change lol.

[@jerry3k]

Great teacher. Kudos!

[@trevorrydalch8959]

I set this up right after setting up my k3s cluster following your HA tutorial. Great work Tim. I then installed rancher, and am struggling to get it to work with the traefik Lets Encrypt certs.

[@IcyTone1]

Did you find a solution?

[@vosproductions37]

1+ year later still gold 🤓!!

[@GrimSpec]

Would it be possible to use this traefik also for services outside of kubernetes ? Like we did in "Put Wildcard Certificates and SSL on EVERYTHING - Traefik Tutorial" ?

[@AfroJewelz]

wonder the same

[@minghou5259]

A cool video for end-to-end.

[@poxin]

Been following along and I have K3S setup from the previous video. One thing I'm confused about is say you have a range of a few IP addresses on the WAN end, do you put these in the load balancer itself (metallb) during the initial cluster setup? Or do you do add a rule in your firewall to port forward 80/443 to the internal traefik IP? Unsure how to get external IPs coming into the reverse proxy properly.

[@thiagomatar7942]

awesome job Tim. Thank. you so much

[@Mikesco3]

This is the first time I've not been able to just follow what you're saying. But that's mostly because I'm not doing kubernetes... However great video

[@TechnoTim]

Thank you, and sorry! I wanted to make a version of my certificate video for kubernetes, like my video for Docker! It's hard keeping up with both! It's not you, it's me!

[@Luckett16]

Can you use this method to create certificates for other internal services that aren't being run within Kubernetes? For instance, Proxmox or UniFi (controller is running on my UDM-Pro)? Didn't know if this was limited to only services running on Kubernetes. Thanks for all the amazing content, I follow your tutorials for a lot of my homelab ventures.

[@TechnoTim]

You can use traefik in kubernetes as your reverse proxy for any service, in, or out of your kubernetes cluster

[@alex.prodigy]

cool video , thanks for all the information made me think if i should also switch from nginx ingress controller to traefik :)))

[@olasojiopeyemi]

Sir you are awesome

[@THEMithrandir09]

What IP does your dns resolve to/router route to when using ingress? The IP of the traefik pod?

[@law1213]

Excellent video Tim, I need to use cert-manager with Let's Encrypt and Private PKI/CA wish me luck. Do you tend to run two separate traefik instances in your cluster one for external and one for internal applications, and for internal is it consider bad practice to use the kube-system traefik provided or should you deploy a separate one for other internal apps as well?

[@Brainpitcher]

Absolutely brilliant.. as always :)

[@TechnoTim]

Thank you!

[@notafbihoneypot8487]

Hey Tim this is a long shot but would you do a video on how to connect TWO HA kuberneties clusters in different parts of the world for true HA Thanks love the videos

[@f1aziz]

Thanks Tim, I roughly followed the same steps. I am using DuckDns therefore I had to use a slightly different path to get the Certificate. Quick question, once you have created the certificate, are these certificate-manager pods necessary to keep around? I guess they do rotate the certificates so we would have to keep them running.

[@kognitiva]

Been running through these kubernetes tutorials, I did this k3s ansible this tutorial then I wanted to install rancher. However, the install of rancher completely obliterates the cluster, making all services unavailable. Cant even get logs from the failed pods. any idea what might be going on?

[@arnepaulsen]

I got it all set up in production mode and even added Heimdall using your sample Nginx yaml as template. Works like a charm. However using HELM to install Portainer is challenging because the the HELM charts create 'Ingress' instead of 'IngressRoute'. Is there a way around the disconnect between HELM Charts and the necessary overrides for IngressRoute that are required to specify Middleware 'default-headers? i suspect whenever using HELM with Traefix we need to download the Chart and manually fix the templates for IngressRoute? Would it be better to reinstall Traefix and use the k8s Ingress instead of Traefix CRD IngressRoute to allow compatibility with HELM? Thank you. You videos are the best.

[@AndrewWilliamsFW]

I've got my SSL certs set up using the OG traefik acme.json, but it's just using NFS for storage on a single raspberry pi SD card - this definitely seems more resilient!

[@yifeiren8004]

Is this k8s cluster running at your home set up? How did you get a external load balancer?😮

[@zoejs7042]

ah terrific. maybe this'll replace my current messy setup using my pi-hole DNS to redirect to metallb services. >.

[@sebastianmolitor4827]

Hi Tim, at first I want to thank you for all your great videos and tutorials. I tried to apply this tutorial, but with only half success. I am using external-dns to automatically register new internal services to my pihole. But external-dns only supports kubernetes ingresses and services. On the github site of external-dns there are some discussions about this. But my knowledge isn't sufficient to understand the solutions there. Maybe you can bring more light in the dark how to set traefik's values or the dashboard ingress to get this running. Thank you

[@alexisbeltranmeza2807]

does anyone get pending for the EXTERNAL-IP ?

[@flesz_]

I think certs are valid for 3 months. How would schedule auto-renewal ?

[@darthweiter7074]

Thank you for your amazing tutorial. I started fresh in the kubernetes world. I used nginx reverse proxy with docker before but want to migrate to kubernetes now. Is there any good solution for using the wildcard zertifikates across namespaces? So i can use it on all of my homelab services. So i could use it for traefik dashboard as well. Or do i need to create my own certificate under my specific namespace

[@stevemulcahy5014]

I was wondering the same thing!

[@TechnoTim]

Either use reflector or see another comment addressing this!

[@KaMZaTa]

In which location of the k8s node do you save your .yaml files? How do you manage them? Do you keep all of them inside a folder on ~/my-yaml-files/ and sync them to a local copy? What's the best practice?

[@ryasan2536]

Hey! Thank you for the greate Videos. I got a problem with your example here, if i deploy traefik, its not creating the traefik-external ingressclass, any ideas?

[@wmchristie]

I landed here because I want to watch Cpt. Jack Sparrow do a tech talk.

[@wmchristie]

Just kidding. I’m a subscriber and your videos have simplified my own process for creating my own home lab. Thank you.

[@mirokko]

I really want video on dns over tls with traefik + pihole. I believe that you can set these dns over tls server on android phone natively and all requests will be forwarded to your pihole server!

[@primeix]

Techno music is a +

[@ryanceki3998]

Thank u man... u r Awsome 🔥

[@TechnoTim]

Welcome 😊

[@Equality-and-Liberty]

Hey Tim, great video. Based on your previous video "Put Wildcard Certificates and SSL on EVERYTHING - Traefik Tutorial" I have created a Traefik container and since then I don't have to worry about certificates cause Traefik is taking care of that. What is de difference between that approach (the previous video) and this one?

[@TechnoTim]

This is in kubernetes and allows this to run HA traefik (multiple instances)

[@ThePC_Geek]

YAS!! Cert-manager!!!

[@squalazzo]

excellent, thanks!

[@xXV1ralXx]

Can you please give me examples on how to add other applications? How would I make Traefik to give Rancher a cert?

[@dylankoke]

Great video! Everything was explained perfectly, although I'm having some trouble. (@31:00) I'm slightly confused about this process. Do I have to have the Nginx test deployment have a LoadBalancer service to get an IP? Then do I tell my local DNS (pihole) that the ingress route match host goes to my LoadBalancer IP? Sorry, just slightly confused. Thanks!

[@TechnoTim]

Local DNS should point at this metal lb load balancer and then traefik will route it to the pod

[@TechnoTim]

Also thank you

[@BensanChong]

@@TechnoTim Hello, thank you for this video it's been really great to follow along with. I believe I'm stuck at the same question as Dylan. to add some detail. in the sample nginx test you give for applying production ssl cert. the service.yml [in the nginx folder] does not have a line for type: LoadBalancer. I'm not sure if you meant for that intentionally? as when I deploy this nginx there is no VIP [External IP] associated to the service if you show svc --all namespaces -o wide. the nginx only has an internal IP. What would be the metal lb load balancer IP you reference in your comment above to which you state to make sure DNS points to?

[@BensanChong]

I've figured it out apologies. You were referencing the VIP created when installing Traefik and metallb produced the VIP. I pointed my DNS to that VIP and voila secured site for Nginx! thanks for this video I've learn so much from it!

[@sachasmart7139]

Incredible.

[@Botio]

I tried this to use two domains on one target with Traefik and RKE2 ... the second domain always shows the CA is provided by Traefik not Let's Encrypt

[@BP-qy2pb]

To make lets encrypted verifying the dns. Does it mean I have to expose k8s ingress to public Network without any reverse proxy?

[@TechnoTim]

No, you do not! That's the beauty of this method, it uses the DNS-01 challenge which does not require your cluster to be public!

[@AlexanderDotH]

Whats the background music? I like that

[@alqods80]

Just more complicated to setup than your other video about automating k8s deployment using ansible, sounds like deploying certs for home-lab environments is an overkill for me

[@BryceTechTips]

How would you find the IP address for the DNS entry you mention at the 31 minute mark

[@AfroJewelz]

just one question: by make dns01 resolver, how to operate cloudflare dashboard when local kubernetes is setup well just like tutorial. i watched the kubectl get challenges for every 2 minutes, status of course is pending ,then open CF dash and i can see my challenges.but when it is over? i was doing the stage steps btw

[@sanrollheiser]

Hi, how are you? I have some problem with set the ingressRoute. The host when try to catch give me ERR_TOO_MANY_REDIRECTS. Any idea? If delete the ingressroute open ok but using the SSL DEFAULT TRAEFIK SSL

[@its_maalik]

Does this auto renew the certificates if they come to expiry?

[@tomklein6540]

Hey Tim, Great tutorial! Keep up the good work :) I've managed to get it all running on my Pi4 cluster. with IPv6 (if someone is interested getting that up and running with RKE let me know) Just one question about the Traefik dashboard though, it seems the middleware takes me to a link with a certificate from Treafik instead of the wildcard. I'm not sure where to adjust that....I suspect its the traefik first setup...from helm... I've changed the ingress yaml to use the TLS wildcard but that only works after authentication. Any idea's? Cheers! ***edit*** Fixed that.... Same problem with the namespace where the certificate is created so Traefik won't be able to fetch it. When creating a separate cert for traefik in the namespace of Traefik and run a helm upgrade it works fine.

[@canislupax]

Thanks!

[@TechnoTim]

No problem! Thank you!

[@vivahernando1]

What if I want to use OCI to provision the lb used by traefik and not metallb

[@daxcor]

How does the 3 month renew happen?

[@damyanmp]

Any ideas if I can pass a challenge with a domain in Google Domains, I was able to generate a token but not sure how to configure the letsencrypt yaml config

[@jonzuka9746]

It is fine and good, but the apache2 secret didn't help me with let'sencrypt of course. Ended up learning more than I wanted.

[@pbolduc]

So what happens when the certificate is renewed? The new certificate will be updated in the secret, but typically the container apps load the certificate at start up and use that and require a reload/restart to start using the certificate. Lets Encrypt certificates are 90 days. Is there anything to automatically restart the pods? Edit: Thinking about my question, it seems traefik may monitor for changes in the secret and load the certificate when the secret (certificate) changes.

[@TechnoTim]

Yes, traefik will load the new cert. It's awesome! It will renew and rotate it autoamtically!

[@gomezsame]

@technotim, can I use Namecheap instead of Cloudflare?

[@TechnoTim]

Not sure! Check DNS01 providers for cert manager

[@MichaelBushey]

If you're already using Kubernetes and Cert-manager (these are AWESOME), please use Istio or another Service mesh. Traefik is horrible.

[@callirgos01]

how does one create a dns entry for 13:18, I am super stuck on this, I cant have my UDM set a static ip / insternal dns record for this IP because it requires a mac address, and the traefik ip given by metal lb is a virtual IP. Any way to fix this?

[@cajuclc]

Thanks for the video. Anyone else having issues where nginx doesn't load on https?

[@mr.engineer-youtube]

Any suggestions how to store cert manager certificates in external volume?

[@TechnoTim]

You don't need to with cert-manager!

[@xavyaly9305]

do you have terraform code to perform the same ? if yes, please share thx

[@m8_981]

8:30 what would i use if i got servers in the cloud? The IP of one of my nodes?

[@RandomGuy-up4bv]

Can you make a video on cilium , cni network driver alternative to aws vpc netowrk dirver

[@madhudson1]

Do you not need A records for the DNS challenge?

[@TechnoTim]

You do not, just your TLD and it will create a txt record for verification!