Filebeat + Elk Stack Tutorial With Kubernetes

Github Repo: github.com/mguay22/elk-filebeat
Artifact Hub: artifacthub.io/
Helm Docs: helm.sh/docs/
Ingress Nginx: kubernetes.github.io/ingress-...
Get my highly-rated Udemy courses at a discount here: michaelguay.dev/udemy/

[@mguay]

Get my highly-rated Udemy courses at a discount here: michaelguay.dev/udemy/

[@awaitingforsunrise]

Michael, thank you so much! You showed us very useful architecural approach: "setup once, run everywhere". With this approach we don't need to be bothering with logging in our microservices at all, because ELK ingests the console output stictly to Elastic DB. It took me some time to get this runned with few corrections of your original files (i.e. "ssl_certificate_verification => false" was required to get logstash being able to connect Elastic and now Elastic requires https connection) + it was needed to bootstrap Elastic's password. Besides it, I was able to get this working and adsorb the logs of all of my microservices in ALL namespaces, despite on ELK lives in its own namespace. Keep it up, nice work!

[@josesantos5517]

You are awesome. Everything worked for me and now I have a complete ELK cluster. Thanks. :-)

[@shaishmuel8087]

Great video, you helped me a lot! It was very clear and straight forward, hope that you will upload more useful content👏

[@po6577]

you make this look so easy!! great thanks!!

[@VinayBedre]

Learnt a lot with your video. Thanks for sharing.

[@nnmbnmbnmnm]

Thanks for sharing, very good explanation. would you please also share what is the log retention policy? like for how many days it will preserve the logs and then purge them?

[@Shreddoctor]

Excellent man! Thank you very much for that brother.

[@usuariousuario5321]

Thank you very much, you help me a lot. How should work to have the logs of the others containers? I just seen the ELK Stack, but a application with logs is not showing

[@nishanthakumara8472]

Great explanation. please upload more videos related to the Kubernetes 🙂

[@mguay]

Thank you, I will

[@OhanSmit]

Thanks dude, this helped a lot !

[@mihowbogucki4928]

Thanks, this was very helpful for me

[@amseshu4u]

@Michael Thanks nice video and explanation.. i would like to impersonate roles and permissions in ELK Stack.. like want use the ELK security feature.. to create different logins for dev_user, dashboard_only_user..etc.. segregating app1, app2..with data1, data2.. etc

[@remram44]

You seem to be a bit confused about how one deploys a Helm chart. The values.yaml is not "what allows us to override the templates", those are the default values to be overridden using the -f option of helm install. You don't clone the chart to install it

[@nitin992vij]

nice and precise tutorial, wanted to know some changes we might have to do when the container runtime is "containerd" like in my case i am using k3s

[@mguay]

Thanks! The setup should be quite similar, you just might have a different path to your container logs.

[@hosseinseyedmehdi4426]

Great video! Thanks

[@user-js9ip7wg9k]

Nice Tutorial , It's helping a lot ! Just to ask , When deploying elastic search using helm chart it automatically creates pv,pvc and storageclass but when actually deployed it only created pvc the pv and storageclass is missing .. which eventually resulted into error "pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.." how to resolve this issue ..

[@ashketchum3255]

Hi Michael, firstly thank u for such a concise lecture but there is one problem that I am facing is that elasticsearch is by default configured to take data at https port while we are making logstash send data on http port, so how do I configure elasticsearch to accept data on its http port as because of this no data is being sent to it. Please help with it as I have checked multiple resources but no one has got a solution to it. It would be great if you'd throw some light on it.

[@JamesAskew-ge9ig]

This is a useful video. At the end I noticed your message property was a json string. Have you looked into parsing that with Logstash, so each property displays nicely in elastic? Also is there a reason you download the helm charts locally and modify the default values.yaml, rather than just providing your own override file during the install?

[@mguay]

Thanks! I agree with your good suggestion to parse the JSON. Additionally, I choose to override the templates directly for simplicity - but I agree with you it makes more sense to override in a values.yaml

[@zongzaili9701]

Excellent Video.

[@nlokesh1986]

Thanks for the video. Can you please help me understand.. 1. How did you setup external-ip to Kibana Pod? 2. Can I implement this solution in AKS? I lokk forward to your response..

[@parthibarajanranganathan9043]

how do i configure my filebeat to get 100's of my pod logs. 100 pod logs -> file beat-> logstash-> elasticsearch. we have nearly 100 pods running in our kubernetes namespace

[@shubhamgawade4466]

Which path do we need to provide in filebeat log path?

[@oliverabrahamhamburg]

Finally I made it to connect to the kibana UI, but only using the forward button in Lens. I still cannot access the UI using my regular kubernetes IP. And I don't understand how you made it to configure the nginx. I would appreciate if you would spend more time for the next tutorial, because it was way to fast for me.

[@TheMyvelmurugan]

elastic search not accepting the empty password connection, have you done any changes in the value file for the auth?

[@jijeshvu8081]

Great Video, but im getting logstash Permission denied - /var/log/containers/ any fix for this?

[@suhasraj9054]

Thanks Michael , whats the cost of running elk stack vs Prometheus within k8 cluster ..which is more recommended?

[@mguay]

Good video idea!

[@manasupadhyay8101]

I have tried but logs are not getting pushed

[@oliverabrahamhamburg]

And kibana is prompting me for a user account. I don't understand which user to take. So I'm unable to get to the Kibana dashboard

[@supermario-dq1uy]

while installing logstack iam getting error Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "logstash-logstash-pdb" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1" ensure CRDs are installed first

[@shivendratiwari-bt3ne]

can we use this is in kubernetes production ?

[@JackReacher1]

couldn't able to run kibana like you did, the current chart of kibana has pre and post actions and it is causing all the problem while installing and uninstalling the chart.

[@raypi2297]

same problem

[@kunoSchlonz]

i tried it with a simple aks cluster (1 node B2ms), but no success. elastic search says insufficient cpu, kibana gives an ingess issue. any idea?

[@demotest-vr4uh]

yes, i had the same issue. Any solution?

[@SureshKumar-my9rr]

Good one

[@UrloDellaNotte]

2024 For those who are getting: "Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "logstash-logstash-pdb" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1" ensure CRDs are installed first" you have to replace "policy/v1beta1" with "policy/v1" (in two different files)

[@demotest-vr4uh]

@mguay with helm install logstash . -n elk Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "logstash-logstash-pdb" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1" ensure CRDs are installed first could u please help with the issue?

[@radupopescu7178]

try like this "policy/v1"

[@demotest-vr4uh]

@mguay * Ingress.extensions "kibana-kibana" is invalid: spec.rules[0].http.paths[0].pathType: Required value: pathType must be specified having this error while installing kibana, could you please help with solution..?

[@heyden2033]

hi @demotest-vr4uh, I got the same issue as well, and its simple to fix it. Basically it omit the "pathtype" inside the value.yaml of kibana ingress section, checkout the other folder (firebase, elasticsearch they all have the "pathtype" in their ingress section), hope this could help you out!

[@ervispojana118]

Hello Michael, thanks for the video , I got stucked at min 14:19, cause as i run in the current directory logstash , it outputs this error: Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "logstash-logstash-pdb" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1" ensure CRDs are installed first

[@ervispojana118]

HOw can i fix it !! maybe there is some error in the helm file, and i have already cloned the repo.

[@alpergenc1832]

you need to change policy/v1beta1 to policy/v1 in logstash/poddisruptionbudget.yml Same for elasticsearch@@ervispojana118

[@demotest-vr4uh]

@@ervispojana118 remove beta1 just keep it has "policy/v1" in all yaml files

[@radupopescu7178]

"policy/v1"

[@syedfaheemabbas4372]

I am trying to deploy on AKS (azure kubernetes service) and getting below error:

[@rickbhattacharya2334]

Outdated video Elastic and KIbana not working !! Elastic is never getting ready in single node cluster ie: minikube, Kibana throwing weird error that i cant even understand. Pls make a new one or delete this as this video is does not work as of 2023

[@mauricekalevra]

yeah same fpr me! old tech videos that doesnt adjust to changes can be such a time waster.