Horizontal Pod Autoscaler CUSTOM METRICS & PROMETHEUS: (Kubernetes | EKS | Autoscaling | HPA | K8s)

🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - [e-mail]
▬▬▬▬▬ Experience & Location 💼 ▬▬▬▬▬
► I’m a Senior Software Engineer at Juniper Networks (12+ years of experience)
► Located in San Francisco Bay Area, CA (US citizen)
▬▬▬▬▬▬ Connect with me 👋 ▬▬▬▬▬▬
► LinkedIn: / anton-putra
► Twitter/X: / antonvputra
► GitHub: github.com/antonputra
► Email: [e-mail]
▬▬▬▬▬▬ Related videos 👨‍🏫 ▬▬▬▬▬▬
👉 [Playlist] Kubernetes Tutorials: • Kubernetes Tutorials
👉 [Playlist] Terraform Tutorials: • Terraform Tutorials fo...
👉 [Playlist] Network Tutorials: • Network Tutorials
👉 [Playlist] Apache Kafka Tutorials: • Apache Kafka Tutorials
👉 [Playlist] Performance Benchmarks: • Performance Benchmarks
👉 [Playlist] Database Tutorials: • Database Tutorials
1. (Helm) How to Install Prometheus on Kubernetes Cluster? - • How to Install Prometh...
2. (YAML) How to Install Prometheus on Kubernetes Cluster? - • How to Install Prometh...
=========
⏱️TIMESTAMPS⏱️
0:00 Intro
1:45 Create EKS Kubernetes Cluster
2:36 Deploy Prometheus to Kubernetes
5:17 Deploy Sample NodeJS App to Kubernetes
7:56 Deploy Prometheus Adapter to Kubernetes Using YAML
12:55 Configure kubectl top Work with Prometheus
18:16 Deploy Prometheus Adapter to Kubernetes Using Helm
=========
Source Code
🖥️ - GitHub: github.com/antonputra/tutoria...
#Kubernetes #Prometheus #K8s

[@AntonPutra]

🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - [e-mail]

[@Samay1205]

There's no stopping this guy.. 👍👍 Amazing explanation.. Thanks Anton 🙂

[@AntonPutra]

Thank you Samay☺

[@anth0o0]

Thank you very much for this video, it's so clear!

[@Firestone752]

Антон, спасибо большое) Жаль что не нашел Ваш канал раньше, наверно сэкономил бы кучу времени)

[@AntonPutra]

spasibo :)

[@anthorosie]

Hey Anton, ty for your great work.

[@AntonPutra]

Thank you!

[@SouthWindzz]

That's really amazing tutorial. Hope you will have more video tutorials.

[@AntonPutra]

Thank you, I will

[@valour.se47]

That was great explanation

[@AntonPutra]

Thank you Valour!

[@timam7580]

You are awesome, Thanks a ton

[@AntonPutra]

Thanks :)

[@santroproin6969]

amazing content man

[@AntonPutra]

Thanks Sandeep!

[@abhinavrastogi9295]

Awesome content Anton, Just wanted to know if the same could be done for Spring boot applications running in OpenShift 3.

[@AntonPutra]

Thank you, I think so, but I never tried on OpenShift.

[@Adam1nToronto]

Thank goodness KZbin has a pause button. All the detail appears to be there, despite a lot of it being glossed over.

[@AntonPutra]

😅

[@arashjavanshour9286]

Hi, thanks for this great content. I deploy the express server but the /fibonacci gives back error and seems like the server cannot handle the GET request. What can be done?

[@ogulcanyagz2568]

Thanks a lot for the video Anton, it is very helpful. I would like to ask something: • What should I do to bind Prometheus with my server's IP instead of working as a localhost. I want to make it always listen instead of appyling port-forward command every time.

[@AntonPutra]

Thanks, you create an ingress or expose it as a load balancer. Typically you would create private ingress, example (no need for annotation) - github.com/antonputra/tutorials/blob/main/lessons/082/example-1/prometheus.yaml

[@iskhak20]

Привет! Возможно ли обойтись без APIService для кастомных метрик? Спасибо!

[@sravu05]

Hi Anton, can we specify multiple metrics in the "custom-metric-prometheus-adapter" configmap? I know hpa supports scaling on multiple metrics, but does prometheus-adpater supports using multiple metrics to scale?

[@AntonPutra]

It's been a while, I think a lot has changed since I recorded that video..

[@pazfelipe]

Anton, your tuto is amazing!!!! I finally could create a custom metric based on http request, however it's not scaling down. Once the pods scale up, they don't scale down to the minimum replicas configured. Would you know why this happens?

[@AntonPutra]

Thanks Felipe! sometimes it takes 5-10 minutes to scale back, you should be able to find the reason in the log if it's stuck.

[@pazfelipe]

@@AntonPutra I've tested it on minikube. Could this be a blocker?

[@tkb12341]

Hi Anton, Can you share the configuration for the below requirement: I need to do the auto scale on increase of user count accessing my UI application and it data traffic is above the threshold

[@AntonPutra]

I have updated version of that tutorial, pls take a look - github.com/antonputra/tutorials/tree/main/lessons/181/1-hpa/custom-metrics

[@user-io8wi7wo7x]

I have problem when i use kube-prometheus-stack to monitor my application

[@tkb12341]

Can we add our custom metrics as below: auto scale on increasing of user count. If so , can you share the Prometheus query and what is HPA scaler needs to be configured ?

[@AntonPutra]

I have updated version of that tutorial, pls take a look - github.com/antonputra/tutorials/tree/main/lessons/181/1-hpa/custom-metrics

[@aditi9636]

Hi Anton, I want to know that do we require amazon aws for this ekstcl cluster creation? Or solely eks.yaml file will be enough to create cluster? Thanks:))

[@AntonPutra]

you need to have aws account, no vpc or anything else is necessary

[@sharhanalhassan7352]

Hey Anton, You'll still remain the best in the DevOps space. A quick one: 1. I want to use the HPA to autoscale different services. I don't want to have each HPA yaml file configured for every microservice. That will be plenty YAML files to handle. The other way round will be to use a single HPA yaml with the "---" separator for all services in a single yaml (the downside is the file will be long) ** Can I create an HPA to target a whole namespace. So all Pods in that namespace should be handled by a single HPA yaml file. Something like: spec.scaleTargetRef.namespace= 2. Outside this topic. I have read you can't use HPA and VPA together unless HPA is using a custom metric. Can I use both HPA and VPA for both CPU and memory metrics while setting updateMode=Off in the VPA so that the VPA only recommends and does not take action on my Pods. Thanks. Looking forward to your response

[@AntonPutra]

To be honest I don't knoe, have you tried to use * in the selector? scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: * Also, if you don't want to copy yaml you can use kustomize (built in to kubectl) or even helm

[@arthurshelby3593]

Hi Anton, great tutorial! I just didn't understand the resource section under rules in values.yaml of the helm chart. Could you please explain? Thanks.

[@AntonPutra]

You mean this github.com/antonputra/tutorials/blob/main/lessons/073/values.yaml#L5-L31? Well, this tutorial slightly outdated due to the new K8s api versions, I'll refresh it soon.

[@arthurshelby3593]

@@AntonPutra Yes, the resource section in the above block of code? Is it somehow related to the metrics-server?

[@AntonPutra]

@@arthurshelby3593 no, in that case we don't need to deploy metric server. That block of code converts prometheus metrics to the Kubernetes API metrics

[@francesn7715]

Hi Anton. This video was very helpful. Have you consider making a Udemy course with more in-depth information on this topic? I cannot find anything like this on Udemy!

[@AntonPutra]

Thanks Frances! Unfortunately, i have a full time job and have limited time to work on all of those ideas that I have including Udemy :)

[@nagasatishvadlamuri8486]

Hi nice .. explanation ..I have one doubt ..in AWS EKS.. Hpa or Vpa which is best for production applications deployment .

[@AntonPutra]

defiantly hpa, most of your applications that you deploy to kubernetes should be horizontally scalable. I rarely see that anyone uses vpa.

[@raviprakash9154]

Hi Anton, thank you for this great stuff. I need a help !!!! I need to modify the default path which is /metrics (property : metrics_path) to my custom path i.e. "/actuator/prometheus" to get all the data. Can you please let me know in which file I can modify the property metrics_path ?

[@AntonPutra]

In the service monitor under endpoint

[@raviprakash9154]

@@AntonPutra Thank you soo much :)

[@hesammohammadi2960]

Hi Anton, I have a problem with this. when load increases hpa works properly, but when load decreases it does not remove additional pods

[@AntonPutra]

The best way to find out the reason is to check logs on hpa controller. If you find an error let me know

[@leozimtub]

All is working but when I try to create my own targets, but it nevers shows up. I did the same configuration of express application, put the label devops on the SM file. And adjusted the selectors to SM hit my Service

[@AntonPutra]

try pod monitors, they are simpler

[@raviprakash9154]

Hi Anton, I need to autoscale for queue message count and for which I don't need groupby clause in metricsquery. metricsQuery: 'sum({})' It is scaling up the pod for message count custom metrics for me when groupby clause is present in the metricsquery. But I don't need group by because now both pods will share same message count and it takes the average. can you please let me know if I have to modify anything else to make metricsquery work without groupby clause ?

[@AntonPutra]

To be honest, I don't really remember all the specifics

[@FuzzPiAndFacial]

Hi ! Why i do not see the targets if i only deploy an app on default namespace ?

[@AntonPutra]

Check service monitor and endpoint labels

[@giingtwigii6873]

👍👍👍👍👍

[@AntonPutra]

☺️

[@JackReacher1]

I created my own custom metrics using prom-client in python which is a counter called "request_count" for my flask app. When I tried to scale with hpa using "request_count"; in the prometheus seeing the "request_count_total" I found that only one pod is serving all the requests; instead of all the pods getting created with hpa. Anton do you know what went wrong?

[@AntonPutra]

Try to find a more descriptive error somewhere in the logs..

[@JackReacher1]

​@@AntonPutra In your first part you use the series query and with the prometheus adapater you don't use any series because "http_requests" have already been implemented by prometheus adapter, right? I just have a doubt here, I would like to know about it in detail. I just implemented your code and I think your implementation is similar to mine. If you go in the graphs section of the prometheus you will find out that your implementation too has the same issue as mine only one pod is handling all the requests. In the graph you will find only one pod handling all the requests and no other pod is handling any request. You can reproduce the issue with prometheus adapter helm chart + your code and check.

[@AntonPutra]

@@JackReacher1 it’s been a while ago, I don’t remember all the details

[@JackReacher1]

​@@AntonPutra I can understand that. Just out of curiosity, if someone told you to complete the above process for a microservice in a company. How much time would it take you to get a dev level code solution?

[@AntonPutra]

@@JackReacher1 it depends how well you know the service. POC is easy, you can come up with working solution for a day or two.. I would say it will take much longer to test and set some trasholds.

[@user-ei2wn4ud4l]

wow

[@AntonPutra]

I'll update this tutorial soon...

[@AntonPutra]

1. (Helm) How to Install Prometheus on Kubernetes Cluster? - kzbin.info/www/bejne/mKi4dnapi6mVg6M 2. (YAML) How to Install Prometheus on Kubernetes Cluster? - kzbin.info/www/bejne/o6WoZaCfnNWbfac

[@AntonPutra]

👉 How to Manage Secrets in Terraform - kzbin.info/www/bejne/aX-TpXqBrNt1mqM 👉 Terraform Tips & Tricks - kzbin.info/www/bejne/bYScZaKLid5lsJY 👉 ArgoCD Tutorial - kzbin.info/www/bejne/sHjRlZqafMZkisU

[@pazfelipe]

Unfortunately this never worked for me 😢😢😢😢

[@AntonPutra]

I'll update this video soon

[@vimalk78]

Can you please re-upload this video without the background music?

[@AntonPutra]

Yes, I’m planning to update it soon

[@vimalk78]

@@AntonPutra 🙏 Thank you !!

[@robinranabhat3125]

It's all good but too high level. I don't know if only I feel like this. Like if you could add a bit more context on what's happening. Rather than just pasting a block of yaml. Especially for beginners out there. But think your content is not meant for beginners.

[@AntonPutra]

Noted. Also, this tutorial is slightly outdated. Newer versions of Kubernetes are using new API versions. I'll update it soon.

[@raviprakash9154]

Hi Anton, I was trying to use external metrics for autoscaling and getting below error. It would be a great help if you can guide me to fix it: the HPA was unable to compute the replica count: unable to get external metric demo/submitter_queue_message_total/nil: unable to fetch metrics from external metrics API: submitter_queue_message_total.external.metrics.k8s.io is forbidden: User "system:serviceaccount:kube-system:horizontal-pod-autoscaler" cannot list resource "submitter_queue_message_total" in API group "external.metrics.k8s.io" in the namespace "demo"

[@AntonPutra]

looks like you need to bind your k8s service account with clusterrole or a role that has proper access