2. Metrics Monitoring: Spring Boot 3 -- OpenTelemetry -- Prometheus -- Grafana

[@giorgidzidziguri610]

Bro, I was struggling with the setup, these tutorial series is hidden gem. thank u a lot

[@linhvudev]

glad you found it useful, thanks so much for your nice words!

[@DanMoldovan]

Thank you for doing these tutorials (I watched most of your videos), your examples are exactly what I needed to get started with OpenTelemetry. Great job and I look forward to your next videos.

[@linhvudev]

Thank you very much 😍! your comment made my day, I really appreciate that 🥰

[@saidmzee4107]

Hello Linh, if prometheus is scraping metrics from collector and the collector has metrics from various services. how can I get the names of the services on prometheus/grafana? so that the dashboards will also contain names of the services

[@prab661987]

Unbelievable tutorials. Not too much theory or too less. Exactly the amount of info needed. and all details are crystal clear. Easy to replicate or improvise for our works. I am using vscode and most of the intellisense is not working. Can you share the extension you used. So that I can try using the same in Vscode. If that doesnt work, will start using intellij. Note: I am doing all this inside Virtualbox ubuntu

[@linhvudev]

many thanks, I didn't use any special extension, and they're just java classes and supposed to be compiled and run the same for jvm.

[@madhand6045]

Hey I have a question while doing this we need a otel collector and promethues up and running on top of java agent which will instrument java metrics and it will be collected by otel collector and it getting pushed to prometheus correct

[@linhvudev]

let's take a look at the diagram, you'll see the arrow from hello-service to collector, it means service "pushes" metrics to collector. On the other side, the arrow is from prometheus to collector, it means prometheus "pulls" (scrape) metrics from collector. Hope it will help you in later videos!

[@madhan5407]

Yeah man got it

[@hiteshkarel]

How you are deciding whether it will be exported to 8888 or 8889 port for Prometheus. Because i have tried first with 8889 it did not work , and it works 8888 . So what is difference between both port? Another problem i want to publish custom metric to Prometheus. Those custom metrics i have created using micrometer as well open telemetry apis. Open telemetry meter counter i can see in otel collectors logs but it is not exported to Prometheus. I have checked in Prometheus UI as well scrap url. Any possible reason?

[@linhvudev]

if you look at the collector in our docker-compose file, you will see it exposed 2 ports: 8888 and 8889. 8888 is the default port that exposes the metrics of this collector itself, you can ignore it since we don't use it in this video. Let's talk about port 8889, this's actually the port that we define for our "prometheus exporters" in otel-collector.yml. We defined "prometheus exporters" at port 8889, so we had port 8889 in docker-compose file, so if "prometheus exporters" is at port X, we should had port X in in docker-compose file (so it's not required to be 8889). And to make a "prometheus exporters" active, we need to add it into a pipeline in otel-collector.yml. With that being set up, now OtelCollector opens port X (8889 for example) for Prometheus scraping data. Then in prometheus.yaml, let's set the scraping configuration to port X (8889). Hope you got the idea and I think you cannot see your custom metrics due to that as well. In this series, we also create a custom histogram metrics, hope it can help!

[@spider279]

what tell us that the otel collector is succesfully running ? why 405 error trying to access localhost:4318/v1/metrics or localhost:8889 ? can you explain me please

[@linhvudev]

Thanks so much for watching the video, you can see the log of OtelCollector to see whether it starts successfully or not. Notice at some lines like "port xxxx" opened for grpc or http. For your case, be sure that: at receivers part of otel config file, you have specified: otlp.protocols.http. By default: http will open 0.0.0.0:4318 which the receiver is going to receive data. Similar to 8889, which is the port that Prometheus will pull data.

[@stevan1533]

Brother I have a problem about the OTPL receiver: 2024-01-10T17:47:26.109-05:00 WARN 63791 --- [trics-publisher] i.m.registry.otlp.OtlpMeterRegistry : Failed to publish metrics to OTLP receiver

[@linhvudev]

Bro, you need to check whether your Otel Collector run successfully or not by `docker logs`. In this video since we've been using http, you should see this line of log `Starting HTTP server "endpoint": "0.0.0.0:4318"`. Secondly, you should make sure you exposed this port to your local machine by specifying port "4318:4318" in otel-collector in your Dockerfile.

[@stevan1533]

The error was a missing 's' hahahahaa Thanks very much brother!