Distributed Tracing With Jaeger And OpenTelemetry In Kubernetes

How to set up and observe distributed tracing with Jaeger and OpenTelemetry in Kubernetes?
#jaeger #opentelemetry #tracing #kubernetes #k8s
▬▬▬▬▬▬ 😳 Sponsor 😳 ▬▬▬▬▬▬
🔗 Robusta: robusta.dev
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.com/4418b87dea48c...
🔗 Jaeger: jaegertracing.io
🎬 You MUST Instrument Your Code With OpenTelemetry!: • You MUST Instrument Yo...
▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendly.com/vfarcic/meet to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Introduction To Tracing With Jaeger
01:40 Push OpenTelemetry Traces To Jaeger
02:27 Robuta (sponsor)
03:18 Push OpenTelemetry Traces To Jaeger (cont.)
06:37 Explore OpenTelemetry Traces In Jaeger
13:25 Distributed Tracing With OpenTelemetry And Jaeger
15:53 Should You Use OpenTelemetry And Jaeger?

[@DevOpsToolkit]

Are you using tracing? If you are, what do you use? Jaeger? Grafana Tempo? Something else?

[@jordangregory9852]

LGTM stack all the way

[@harishahmad]

ELASTIC APM Support open telemetry?

[@senpai70]

I used Jaeger in the past but now testing APM with Elastic combined with opentelemetry collector. Also the Grafana stack is on my radar. There is a good demo application from OT with an helm chart that deploys everything to have a working environment in minutes, included a load generator so you don’t have to click around and simulate user activity 😉

[@harishahmad]

@@senpai70 i am looking for the same, i deployed APM without open telemetry

[@senpai70]

For me otel collector is a must, can feed into any compatible backend so you can switch between them. I plan to feed the same data to the Grafana stack and compare… all I need is time

[@swarnapriyajena8042]

You are amazing. Yes, kindly advise that we use Opentelmetry solely for request tracking. Alternatively, we might use it to replace Fluentd's log aggregation and routing functions with Logstash.

[@juanitoMint]

I'm using a mix of Jaeger and OTEL, centralized collection from several clusters on a single back-end running ElasticSearch (hot-nodes/warm-nodes) using elastic-cloud operator

[@kevinyu9934]

Thanks for making such amazing contents. It rly broaden my horizon. Cheers.

[@DevOpsToolkit]

My pleasure!

[@alvarotorres3529]

Great video as always! I think it would be awesome if you did a video exploring observability in grafana stack (with grafana cloud, for example, which has a free tier). There, you have Prometheus (metrics), Tempo (traces), Logs (Loki), and Grafana (visualization) all working together. Actually, I think you can correlate all of them and look up traces, metrics and logs through the same dashboard. This is a topic I'm personally very interested in. Thanks for your work

[@DevOpsToolkit]

Grafana Tempo is already on my TODO list :)

[@jeanchindeko5477]

Yes, please what are the alternative OSS tracing visualization solutions or commercial soluout there

[@trantrunghieu1698]

Great video. If jaeger does a fair job of extracting traces info and reliable then it's good to me, cause it look quite easy to use.

[@ParthKhandelwal-jj5qb]

Yes we want to see comparison

[@PaulGilowey]

Thanks for this one too, Victor. Is there a down side to instrumenting applications by using istio sidecars instread of adding it throughout my codebase? Also, did you get around a video on Jaeger alternatives? I'm specifically thinking about Tempo because we use Loki and Prometheus.

[@DevOpsToolkit]

It all depends on the level of instrumentation you need. Using external instrumentation like the one from Istio is a no brained. Use it. However, that gives you the information that does no go deep. It will not, for example, track what's going on inside the app (e.g. duration of execution of a function). So, start with Istio and, if you need something depper, add instrumentation to your code. Also, there are many options to automatically instrument your code so that's an option as well. As for tempo... It's great and I have it on my to-do list for an upcoming video.

[@christopherreimer9951]

Congrats on winning devopsdozen award well deserved. I wanted to ask if you could compare jaeger to SigNoz or generally go into SigNoz it looks promising, as an OT solution for all 3 Logs, Metrics and traces. And im still not sure if i should use it, and would like to see an unbiased opinion. (of course the creators are not going to tell you about Potential weaknesses) would greatly appreciate it. Christ bless and save you

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@shalomcohen122]

An important aspect that was not mentioned here is should you have traces for everything constantly? Should you enable when troubleshooting? Should you use samples? Maybe trigger upon errors? Sending and keeping all the tracing data for a big production system can be very expansive, add overhead and is not necessary most of the times.

[@DevOpsToolkit]

Depending on the size of your system, you probably want to send samples instead of all traces. One out of a hundred or a thousand… You should not enable it only when troubleshooting since that might be too late, at least when running in production.

[@hossamshehab8301]

Observability is better when combining metrics, logs and traces. So better to choose Tempo

[@DevOpsToolkit]

A combination of those three is indeed a must.

[@dirien]

Yes, that’s why I prefer the LGTM stack by Grafana! All in one! I formerly used also Dynatrace as it’s offers also a aggregation of different sources for a better insight

[@tomjosi742]

great video, very enlightening to the questions i were getting. i have one question though, when i run docker compose up i only see jaeger-all-in-one on the UI. Running the curl or anything doesn't make the services appear on the UI. Am i missing anything?

[@DevOpsToolkit]

I tend to run it in Kubernetes so I haven't tried it with Docker Compose 😔

[@AnkitKumar-yi1ir]

I have 2 basic questions 1. Let's suppose we have implemented the instrumentation of code and we have not deployed the jager maybe for 2 months, in that case, what will happen to the data/traces of opentelemetry? How we are storing and managing the data in open telemetry. And if the system is huge and we are keeping the data, what best practices we can implement for managing storage? 2. How is Kiali different from Jager? If we have istio implemented, then Jager/opentelementry is required?

[@DevOpsToolkit]

1. OpenTelemetry is an API spec. It does not store data. Jaeger and others are data stores. So, if the data is not stored anywhere for 2 months, you cannot retrieve it. 2. Istio is networking solution that does not add trace IDs to requests. You need to add those IDs or use a tool that does that for you. Without them, you cannot connect different requests into a flow. Both kiali and Jaeger have visualizations that show you what's happening with network traffic. However, jaeger is also a data store that allows you to query past data while istio does not store data beyond short period.

[@DaljeetSingh1]

awesome!

[@WeRRa89]

It would be interesting to compare to grafana tempo being one of the newest additions to the cncf in terms of tracing. In addition to that it would be interesting to know if there are any differences in using different visualization tools vs using Jaeger's built in UI.

[@DevOpsToolkit]

I'll explore Grafana Tempo in an upcoming video. A small correction... Tempor is not a CNCF project (as far as I know).

[@WeRRa89]

@@DevOpsToolkit you're right, it's not (yet). Nevertheless it's still a new contender that's interesting to examine!

[@DevOpsToolkit]

@@WeRRa89 Oh yeah. Tempo is awesome and I will certainly make a video about it, I just can't give the exact date (at least not yet). It's coming...

[@arozendojr]

Can you answer a question, is it possible to use jaeger + istio, for every request and response event of each microservice? automatic without changing microservice/pod code? How can I look for the configuration I should do?

[@DevOpsToolkit]

You cannot not change the code of your apps since that code needs to extract trace ID from incoming requests and inject it into outgoing requests. Otherwise, without that ID, Jaeger cannot track requests. That being said, there are many libraries that make that instrumentation very easy and require only a few lines of code. There are also tools that will instrument code automatically so that you don't have to write those lines.

[@avanthaditya8596]

Very neat and clear explanation. Would you please compare jaeger with other commercial APM tools?

[@DevOpsToolkit]

I will not be able to do that in the coming months because I will be focusing on CNCF projects. It's related to something big we're starting in early February and will occupy most of my time. More info is coming soon. The downside is that I'll probably have to postpone non-CNCF content to some later date :(

[@robarros21]

very good

[@tomasferrari92]

Hey Viktor. If I'm using istio, do you think it is still necessary to instrument the code?

[@DevOpsToolkit]

Yes. Istio will not provide information that is available only inside the code.

[@tomasferrari92]

@@DevOpsToolkit thank you 🙏

[@cresento]

Can u please share an architecture diagram focussing on collector gateway with jaeger plus elastic backend

[@DevOpsToolkit]

Unfortinately, i haven't used any of Elastic products for a while now so i might not be the best person to make such a diagram.

[@hrishikeshlimaye3747]

With Istio, how the setup will look like to get traces in jaeger through open telemetry? Do we still need to instrument code?

[@DevOpsToolkit]

Those are different types of traces. Instrumenting your apps gives you traces from inside your code (e.g., a specific function) while Istio or, to be more precise, Envoy gives you traces observable from outside the app (e.g., a single request).

[@mechrono9555]

Please showcase open telemetry collector exporting data going into tempo and grafana in the next video, in the middle of setting this up with helm charts

[@DevOpsToolkit]

It's already in my TODO list. Bumping it up closer to the top of the list... :)

[@user-ij5np7iu5d]

i think it would be awesome if you explore prometheus vs jaeger

[@DevOpsToolkit]

I don't think those are comparable. Prometheus stores metrics and Jaeger stores traces. Those are very different types of data.

[@PhiLongAnGia]

Hi, thanks for your video. Currently, I'm using Datadog to monitor and trace the request in my system. Datadog agent can inject traceId when calling HTTP from Client to API ( A ), but in API (A) we make the call to API (B) via TCP transport. In this case, the DD agent cannot inject traceId in API (B). Do you have any solution for injecting traceId in this case?

[@DevOpsToolkit]

You need to modify the code of your app for that. Alternatively, depwnding on the language, you might be able to instrument your app automatically.

[@fatihbaltac1482]

Great video about Jaeger, thanks! Could you compare Jaeger with Zipkin?

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@johnfli]

TEMPO, we want to see OTEL+Tempo by you, Viktor! Everyone in here is thinking of this, right?

[@DevOpsToolkit]

Will do. It's already in my TODO list. I just need to push it closer to the top :)

[@RockTheCage55]

Definitely compare jaeger to other non cloud based solutions……thanks

[@ricdotdev]

It's pronounced "ye-ger" (well not exactly but that's the closest). Basically, the same way they call the mechas in Pacific Rim.

[@DevOpsToolkit]

That was silly of me, but not the first time I misspronounce something. I need to be better at checking pronunciation. Thanks for the tip.

[@arcanernz]

Is this equivalent to aws x-ray?

[@OOpSjm]

Equivalent?

[@DevOpsToolkit]

Yes it is. Both serve a similar function.

[@LionelMarks-bj6ku]

you didn't talk about how Jaeger used to have it's own tracing format before OTEL. wouldve been useful to understand the history

[@DevOpsToolkit]

You're right. I should have mentioned that. I was so focused on OTel (this video continues where the previous one about OTel stopped that I forgot to talk about Jaeger's past history. My bad...

[@lamnot.]

But ofc Prometheus vs Jaeger.

[@DevOpsToolkit]

Those serve different purposes. Prometheus stores metrics while Jaeger stores traces. So, it wouldn't be vs type of video but rather what are the scenarios in which one should be used over the other (what is what type of video). What do you think?

[@lamnot.]

@@DevOpsToolkit Sounds right for now, as we see eBPF based Metrics, Tracing and Profiling tooling changing these differences.

[@DevOpsToolkit]

You mean the format or the storage?

[@lamnot.]

@@DevOpsToolkit The format will likely standardize around OpenTelemetry, and free up the choice for any storage option. A single window observability tool combining metrics, trace and continuous profiling could be worth comparing with perhaps?.... Though I don't know any specific project doing this.

[@DevOpsToolkit]

@@lamnot. Personally, I'm not sure there is the need to combine where those are stored. In the past that proved to be inefficient (e.g., Elastic). What matter much more is to have a tool that will present those three in a unified way. That's where Grafana (including Tempo) come in.