Server Monitoring // Prometheus and Grafana Tutorial

[@christianlempa]

Please be aware: update to the Git Repo: github.com/xcad2k/boilerplates I now separated the docker-compose files for Grafana and Prometheus in two separate folders!

[@giftcp82]

why. is there any advantage to doing this or were there challenges with the first configuration.

[@Trains-With-Shane]

@@giftcp82 In case you wanted to set them up standalone. Not a problem for somebody who's familiar with docker compose YAML files but for somebody just starting out it could get confusing having it all together. This way it narrows the focus and doesn't get everything potentially confused for the new user.

[@AbdulRahmanKayaliIbnMukhless]

What is the difference between grafana/grafana and granfana/grafana-oss as your updated Git Repo refers to the latter while video refers to the former?

[@mridulranjan1069]

With time any updates like this will be harder to find in the Comments section. I was lucky I found just when I was wondering the video shows a combined docker-compose.yaml but the actual repo has them separated. Maybe update the main Description section with a dated update about the repo? You know, just like a code description update :-) Anyways, loved your video so far, the presentation. Great Job!

[@mehammered]

The exporters folder seems to be missing also. Edit: Its in the dir is the prometheus folder not within it.

[@Psyt0s]

Man, I can't thank you enough!! all your videos are so easy to follow and actually understand what I'm doing!! You are awesome!!

[@christianlempa]

Thank you so much :)

[@Flako-dd]

Awesome man, a youtube guide that doesn't start with notepad on windows, but immediately gives you the github repo. Love it, great guide!

[@christianlempa]

Thank you so much! :)

[@jacobcrisman6834]

I don't know why I try to go anywhere else when I need something installed in Docker. These videos always give more information than what is needed and everything is explained very well

[@christianlempa]

Thank you! :)

[@IgorKuts]

Thank you, Christian, for this ultra-high quality content. Learned a lot. One thing tho, as for me, frequently i found it difficult to keep up with your pace.

[@packetattack-netsec]

Great tutorial, thank you! For any beginers out there. if your like me.. im trying to learn YML syntax... Linux and docker/portainer all at the same time.. Make sure your syntax in your prometheus.yml file is correct or else it wont start... I also had to identify the user: root in my stack editor for prometheus so that /etc/prometheus/prometheus.yml could be read.. im sure thats not secure.

[@auroraveon]

the volumes that your container accesses, run this command on them: sudo chown -R 65534:65534 prometheus_example/ replace "prometheus_example" with the folder name that you provided to your volume(s) 65534 is the user and group id of the user/group that runs prometheus in the container. for me, grafana also had permission issues in the logs of the container, too. the user and group id is 472 so you'd replace "65534" with "472" in the command above.

[@wildflowers465]

Hey Christian, another amazing video! I set this up in my home lab, and it works great.

[@xx482]

Before going to watch this video .. i really appreciate you for making this video .. i was struggling to learn the basics of Prometheus and grafana , you really helped me by putting this video.. Thx in advance

[@christianlempa]

You're welcome!

[@f669ciycoycutfiyciyxutxohcirdi]

Great tutorial, which I used today. It also might be helpful to show max cpu core utilization and overall cpu utilization alerts. While I was trying to setup alerts for home lab, I discovered that Sys Load is showing 170% and CPU Busy 85% when top shows about 10% utilization when I go to the 6 core debian virtualbox node for dashboard 1860 Node Explorer Full. Those stats on that dashboard seem misleading to me, and looking at another dashboard.

[@christianlempa]

Oh yes, that's another project I want to look at next year :D

[@daiben2004]

He Christian we need to complete this course with Prometheus Alert Manager and Grafana Alters

[@christianlempa]

That's a great idea, unfortunately I can only get to it somewhere next year, but I'm looking into it

[@daiben2004]

@@christianlempa Thanks 🫥🫡

[@hikmatullah9463]

It's was a nice video about Prometheus and grafana

[@YouTubers-rj9xv]

Bro excellent teaching

[@christianlempa]

Thank you mate! :)

[@qoutwest]

Great video! Going to try and set this up now that I have my Proxmox server up and running

[@christianlempa]

Awesome! I'm currently working on getting this running on proxmox as well 😁

[@keyshine5769]

@@christianlempa can you please make another full video when you got your proxmox up and running especially monitoring on LXC and etc. Im sure it will benefits alot of us!! 😭😍

[@christianlempa]

@@keyshine5769 I guess that I will replace Proxmox at some day and focus more on Docker than LXC, there will be a big change in my Homelab next year 😁

[@George-zm4iu]

How have i only just seen your channel now! Awesome video, i enjoyed your ad too lol. Sub gained :)

[@christianlempa]

Awesome, nice to hear you're liking the content 😁

[@leboncoinyca4704]

Great video as usual. Thanks !

[@christianlempa]

Thank you :)

[@FrederickAlvarez_]

I love your content, you the teacher I never had

[@christianlempa]

Thank you so much :)

[@soukainaboualy912]

Thank you so much, it's so understandable nd clear

[@locosporasturias]

Amazing, thank you very much for your time

[@alemran7480]

Thank you so mush ❤ Fantastic tutorial 👏

[@christianlempa]

Thank you so much ;)

[@freshlix9554]

Prometheus is cool and stuff, but I would prefer VictoriaMetrics as a drop in replacement - because it's case studies seem pretty neat..

[@cezwitsteez]

thanks for this tutorial man, super helpful!

[@christianlempa]

Glad you liked it!

[@Stingray7423]

The best!

[@ramyasreetejo]

content of high standards!!

[@sammanakkuon8581]

Thank you for sharing

[@UltimateKeyboardHero]

I love your two recommended dashboards! was using statefulset daemonset metrics board before, but that's so bad compared to the node exporter full!

[@RomanF-h9t]

thank you for your work

[@christianlempa]

You're welcome :)

[@sarmadahsan869]

amazing, Help me a lot. Thanks

[@christianlempa]

You're welcome!

[@mrshafee95]

thank you bro very awesome thank you again thanks a lot

[@christianlempa]

Thank you! 😉

[@christianajeitoh6533]

Thanks very much for the content. It's very informative. I have a question. If I have n nodes in my cluster. will node exporter give metrics for all of these node machines OR do I need to run node_exporter on each on these nodes

[@christianlempa]

you have to run node_exporter on all nodes as far as I know

[@alpachino468]

I wasn't able to get through the entire video yet, but please correct me if I'm mistaken: Grafana and Prometheus are better installed on a local server within your network for protection etc, and Node Exporter/Cadvisor should be installed on remote nodes that you would like to monitor?

[@ernestodepesz11]

I got an error during running all staff: Error response from daemon: path / is mounted on / but it is not a shared or slave mount any ideas?

[@mohammad-aminebanaei886]

awesome thank you

[@christianlempa]

You're welcome

[@akyumurkov]

Junior sys admin (me) wet dream! Thank you!!!

[@christianlempa]

You're welcome 😊

[@samdhillon8325]

Nice tutorial as always!! Thank you Christian. I have tried to follow your steps to setup Prometheus but for some reason its is not reading the configuration file "prometheus.yml" from the host instead reading it from the prometheus container which i don't want. Can you please help me figure out what I may be missing here :-)

[@cybersecurehacks]

Nice. I am gonna implement this tomorrow in my organization. Also is there a way to manage esxi storage from some 3rd party tool?

[@christianlempa]

Awesome! Unfortunately I haven't worked with esxi storage lately.

[@mohammadhadi8142]

Perfect

[@christianlempa]

Thx!

[@kingsadmin]

nice one!

[@christianlempa]

Thx!

[@NemesisKo1]

awesome

[@mateusjunior1937]

Hi i'm a big fan of your channel, is it possible to make a tuto on zabbix and grafana, just as example

[@christianlempa]

Thank you :) maybe in the future!

[@yurafinzi]

Hello Sir, i'm using container (LXC) to put database, can I install mysql_exporter in another container to monitor my database container? or does it have to be in my database container? thanks

[@firinggamers3217]

hey im having issue whilt trying to uncomment the node exporter and cadvisor once i uncomment them and restart the promethues container its giving me an error Error loading config (--config.file=/etc/prometheus/prometheus.yml)" err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 19: did not find expected key

[@firinggamers3217]

found the issue i was using non arm on arm i guess (node_exporter)

[@devshah4030]

Awesome tutorial. How do you use APIs as a source for feeding data in to Prometheus?

[@christianlempa]

Haven't done it before, but I'm sure you need to create an exporter that propagates your API metrics to prometheus.

[@devshah4030]

Thanks @@christianlempa somehow Grafana has custom plugin to ingest Dynatrace feed suing their APIs. Not sure how it will persist the API response or if it would be direct feed to the dashboard in realtime.

[@ОлегАрбузов-й5у]

thanks a lot, but what about domain name to al this staff?

[@NeilMendham]

I'm trying hard to get this set up and move onto the grafana bit of the video, but I can't get past the error finding the prometheus.yml file: no such file or directory. I have checked spelling, checking permissions, changed permissions to everyone RW, the container still won't start with the --config.file command; it doesn't find the file even though its there!! Frustrating the hell outta me!

[@larssenYT]

I had the same issue, the following worked for me: 1. Connect in ssh as a user (larssen in my case) 2. Create a folder prometheus in /home/larssen 3. Put the prometheus.yml file there 4. In the docker-compose, change the folder accordingly. In my case, for the volumes: - /home/larssen/prometheus:/etc/prometheus Hope this will helps!

[@bluerev8353]

i have error like : Error reading Prometheus: An error occurred within the plugin. anybody can help?

[@christianlempa]

Maybe join our discord and share your configs :)

[@zulhilmizainudin]

Do you have any recommendation for the volume sizes use for Prometheus and Grafana to store their data? I'm planning to run them in a Raspberry Pi. I'm afraid it will take up the whole SD card space. In other words, what's the ideal volume size for Prometheus and Grafana containers with default 15 data days retention?

[@christianlempa]

Hm, my experience is limited here, unfortunately

[@aswanthpp3026]

How can I securely add an remote server?

[@yolob2116]

can you please show the cadvisor exporter configuration for datasource. I am not getting any metrics

[@christianlempa]

I think that should be in the boilerplates repo on GitHub, if it isn't , hit me up on Discord!

[@Raf99]

You look and sound like AntVenom

[@MrScX]

Great video! Really appreciate it. One issue I'm facing is with node_exporter. I'm on Mac and I keep getting this error with node_exporter on startup: ERROR: for node_exporter Cannot start service node_exporter: path / is mounted on / but it is not a shared or slave mount Any idea why this is happening?

[@hanweijie8092]

I'm having this error too.. any solution for that?

[@AwOiSoAk]

In my case I had to change the node_exporter volume to volumes: - '/:/host:ro'

[@martiiimsilva]

@@AwOiSoAk I love u

[@ctcrnitv]

you probably already know this and just saying it out of convenience but "etc" doesn't mean et cetera

[@leonelasetto3147]

a donde tan peinado mi amor?

[@finnteuwsen3789]

How can i protect prometheus on a vps?

[@christianlempa]

Sure, I would just be careful to not expose the port and connect through a secure network like a VPN.

[@1over137]

Do yourself a favour. Set your retention policies or reduce down jobs 'before' you start.

[@Symbiiotic]

This video gave me a headache.

[@svenle7787]

/etc/prometheus/prometheus.yml: yaml: line 24: did not find expected '-' indicator"

[@cyberlord64]

Absolutely invaluable. I have gone through a few of your other videos as well, and everything is essentially easily digestible essential knowledge that would otherwise take me weeks to search and figure out on my own. A huge plus is that the scripts and snippets provided actually work out of the box and everything is explained on an almost line by line basis. Thank you for your amazing work

[@adeelhashmi145]

wait what? invaluable?

[@bradleydiggs]

@@adeelhashmi145 It's weird that it means what it does (with the prefix in- that normally means the opposite), but it actually means "incredibly valuable" or "more valuable than valuable"

[@user-se5xg3vz8p]

@@adeelhashmi145 priceless

[@iaamnew2060]

To all that have the /etc/prometheus/prometheus.yaml: is a directory issue: Problem is volumes: ./config/prometheus.yaml: it's trying to copy in the prometheus.yaml file from wherever this is setup to be but that file doesn't exist so it thinks it's a directory To fix this just make the config file manually somewhere and get an absolute path and just replace that line with the absolute path and then delete the stack and deploy again and it should all work.

[@iaamnew2060]

Not working, when I setup prometheus with config it keeps giving that error /etc/prometheus/prometheus.yaml: is a directory"

[@ryanbell85]

If you are using Ubuntu 22.04 and missing memory stats in grafana to monitor your docker containers, you'll need to change the version tag on cadvisor to: v0.44.1-test

[@auroraveon]

for anyone who cannot get the prometheus or grafana container to start or they keep restarting (which is what happened to me), use the details below: the volumes that your container accesses, run this command on them: sudo chown -R 65534:65534 prometheus_example/ replace "prometheus_example" with the folder name that you provided to your volume(s) 65534 is the user and group id of the user/group that runs prometheus in the container. for me, grafana also had permission issues in the logs of the container, too. the user and group id is 472 so you'd replace "65534" with "472" in the command above. this issue occurs due to the user inside each container not having the correct permissions set. to my knowledge, most people here shouldn't have a problem if you followed the steps provided in the video correctly (which i didn't and i changed a few things with the volumes which is probably why i had this issue)

[@trekydeky]

Thank you so much for this.

[@auroraveon]

@@trekydeky glad this helped you!

[@edisvila]

Great video as usual. There's just one thing though that I'm curious about and that is missing here. Notifications, for example when a service goes offline or when a specific event happens. I've seen that Grafana has an Alerts feature. I would really appreciate if you could make a video on just something like that for real life scenarios where a notification is useful

[@christianlempa]

Such a great idea! I'm planning a new video about that, but that's a bit down my list for next year.

[@edisvila]

@@christianlempa Looking forward to it :)

[@1Chitus]

Great video ! Just one tiny question: I intend to install Prometheus and Grafana on a RPI4 with 2GB of RAM. Is it enough to monitor about 5 containers and system metrics ?

[@mohamedhaneef6379]

Awesome!. I would like to work on a solution that would visualize various metrics from Palo Alto firewalls. I saw an exporter for Fortigate firewalls. Could you please suggest a way forward? How would I customize an integration option for Palo Alto firewalls? Thanks:)

[@adeelhashmi145]

remove the host if you get any error in cadvisor

[@ShopMeeseeks]

Hey, I'm being tasked with setting up a monitoring tool for my company which will be configured to scrape data from many servers and applications. Do you think this method is the best to move forward with or is there another you think would be better suited? Please not trying to use the most inexpensive option! Thanks :) and your videos are awesome

[@Goingto-tech]

Great video ! Just one tiny question: I intend to install Prometheus and Grafana on a RPI4 with 2GB of RAM. Is it enough to monitor about 5 containers and system metrics ?

[@Naz-pk4ll]

would be great if you could update the video. volume mounting for prometheus yaml config file seems to no longer work.

[@mohammedsnoussi8782]

Grafana dashboard layout displays all servers as IP addresses which is difficult for us to keep track of and we prefer to have it as a hostname display in the drop down. Any suggestions on how to get that done would be very helpful.

[@0917hubing]

今天边学习，边自己服务器配置了监视系统，谢谢Christian Lempa。

[@ShijuJose-o6x]

Hello Christian, Thank you for the fantastic video. I have a question about Prometheus. The cadvisor and node_exporter targets are not showing up on the Prometheus page for me. I can see that the web interface is functioning when I log into node_exporter.

[@christianlempa]

Thank you :) maybe check the config and docker network routing, otherwise maybe someone on discord can help you

[@jiri_kuzel]

Awesome introduction into Grafana using Prometheus!

[@tirushv9681]

Happy to see this video particularly by "The Digital Life" Thanks sir cause i have seen many videos already but here we can get exact points to remember and its basically on point

[@christianlempa]

So nice of you!

[@srleb3462]

Christian, thanks for great informative video. Would be great to see one video with more in depth steps by step instructions.

[@christianlempa]

Thanks! I hope to make more videos about it next year

[@joshuaplouffe1439]

Could you do a video on how to set up alertmanager on top of this setup? Its a missing piece to this AMAZING tutorial!

[@christianlempa]

Great suggestion, however I'm still working on so many projects. But at some point I'll make a video about Grafana Alerts maybe, we'll see ;)

[@playeronthebeat]

Hi! I followed your tutorial but unfortunately, I can't seem to capture the metrics in Grafana. For some reason I can't see prometheus' metrics in Grafana. They're available to me, if I go to the prometheus WebUI but not from within Grafan :/ When I add the datasource, Grafana tells me that it was successfully querying the Prometheus API. Yet, I can't see anything from Prometheus :/

[@rafabrand_]

Pls, i have installed haproxy in 1 server and i need to get The information from Prometheus I installed Prometheus in other server too How can i get The information from haproxy server?

[@lynic-0091]

This isn't really a Prometheus and Grafana tutorial, more like a getting started / installation.

[@therus000]

Thanks. nice video i upload the dashboard 1860 but can't see the CPU metrics all metrics work on dashboard except the CPU, i try to look for metrics in prometheus, they appear there

[@ComputerworxVideo]

Can you please make a video about Prometheus, Grafana and Sophos XG Logs Monitoring ! I would like to be able to see multiple XG firewalls Critical and alerts in one Database interface or even 1 firewall.. can this be done? btw you promised more Data monitoring setups on your channel but I have not seen any.

[@TechSignn]

Do you have any video how can i configure Prometheus to send data collected via email as a CVS or PDF file ?

[@silverace_71]

For some reason I cant get cAdvisor, or grafana to go up. When I check the logs, it's just saying | standard_init_linux.go:228: exec user process caused: exec format error | I am scared and confused, can someone please help? (I am using a raspberry pi4 on 64bit raspian if that helps)

[@5UPRAH]

I've got both up and running but not talking to each other. I added that "Node Explorer Full" but all my displays only show N/A. I can choose the data source at the top but node and host I can only select "None"?

[@averagedev7768]

Used cockpit for the last 2 years. I am very happy how it works. I have 7 servers in total i have to manage, and cockpit allows me to do just that by logging in on one single server adding others to there and watching there progress

[@christianlempa]

Nice, yeah cockpit is pretty cool, I also use it sometimes ;)

[@martinlortie]

Yep after turning on Cockpit this is exactly what I needed, but if you have more that one server with multiple instances of things then I can see Prometheus/Granafa shine.

[@haralc]

Is it just me or prometheus is a bit strange? I wrote an application for testing that increments a counter by 1 every 5s. That said, prometheus can only show me .2 when apply the rate, even if I set the "step" as 5 (seconds)... this is quite annoying because there's no 0.2 request per second ... like you cannot split a human into smaller pieces.

[@innocospaceco-working9346]

Thanks for the excellent tutorial. I created the docker-compose file according to your steps, but both my garfana and prometheus containers kept restarting every 2 secs. I don't know why it is doing that. I increased memory and CPU on my host, and it didn't make a difference. I even created the prometheus.yml file in the prometheus directory, and it didn't make any difference. Any idea why this is the case? Thanks for any help.

[@christianlempa]

Thanks mate! Regarding your problem I would check the log files why the containers are restarting, usually you should see an error here.

[@Zero-qs8uq]

Hey! Thanks for your video 😄 I have a question: There is a way to use Prometheus Cloud + Grafana Cloud without self-hosting it, if yes, then how? Thanks!

[@shafira_yulia]

Thankyou for your explaining Sir, I have a problem, I wanna make one dashboard for one server, is it possible? because all the server lists that I have inputted into Prometheus.yml go into the monitoring dashboard automatically, I want the dashboard to be separate for each server

[@abhisarbharti1920]

I am having difficulty installing prometheus. I am using windows system and the script in portainer is unable to find the prometheus.yml file. Any one else is having the same problem?

[@scholziallvideo]

hi, perfect video. i use prtg monitoring for the servers etc. Primary windows servers and vcenter hyper-v etc.

[@jos1532]

Hello thanks a lot for your video it is really demostrative :D, Sadly I've ran the docker-compose file and I get the following error: Error loading config --config.file=/etc/prometheus/prometheus.yml)" err="open /etc/prometheus/prometheus.yml: no such file or directory". I made sure of creating the file in /etc/prometheus but I had no succes, did you face some issue like this? Thanks in advance :D

[@christianlempa]

Hey mate, sorry but I guess that's because I changed the /etc/prometheus folder to /config on the host, I've updated the github repo again, you might try with the changed docker-compose again and please tell me if it's now working! Thanks :)

[@jos1532]

@@christianlempa beautiful thank you a lot, that works fine :)

[@parkiel54]

@@christianlempa Hey, I could be doing something incorrect. But I am getting this error on your current prometheus docker-compose.yml

[@erfanreza8789]

Can I add different remote (linux servers) using the above implementation? What is the processor? Thank you :)

[@-a6833]

I'm running into the weirdest issue I've seen in a long time, I can copy paste your docker-compose file word by word, and yet the container keeps restarting saying it cannot find the prometheus.yml file in /etc/prometheus/ even though I can clearly see it's there, also it doesn't matter if I mount a different volume to it (I figured maybe permission issue?) it will fail with the same error message.. I've tried dropping into the container to see if the volume is properly mounted and the file is there, but since it's constantly restarting that's not possible. Also, I've tried removing the config file start command to see if it will start without one, which only causes it to search for a config file in it's default directory which for some reason it also cannot find, very strange. But considering nobody else here seems to have this issue I must be mucking something up Basically it cannot seem to see the config file Edit: It seems to work now when I map the .yml file directly, and omitting the --config boot command, instead of mapping just the location /etc/prometheus/

[@SravanKumar-bi9mw]

very useful and Great video - I have done same process and dashboards got visible:-) But i got additional requirements to monitor ,Linux OS version. Which i am struggling and not showing with this ID 1860 ,Moreover, I am using node exporter +Prometheus data source. Can you please suggest on this.

[@brianachenbaugh1470]

Hi christian, I am new to this and we need to set something like this for our F5 devices. They have their own plugin called Telemtry. Have you ever built anything like this with F5 or similar?

[@gecmisinisevenadam3971]

Issues with cgroups. Cadvisor don't want to recognise that I have cgroup2fs on my ubuntu device.

[@conradtaylor29]

Hey Christian, this was a fantastic video and I learned a ton. I was wondering, how did you setup Portainer?

[@christianlempa]

Thank you! I think there's a tutorial for portainer on my channel :)

[@conradtaylor29]

I just ended up adding Portainer to the docker-compose.yaml file and this appears to work.

[@lcbdias]

Amazing content!! I'm getting the message: Get "cadvisor:8080/metrics": context deadline exceeded Any thoughs on that? My cadvisor container log is: Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: operation not permitted

[@mtbjonny9502]

.. and how to add multiple devices in the Grafana dashboard using the same dashboard (there is a pull-down menu) i haven't seen a video that explains it

[@daiben2004]

Nice Explanation, It will be more beneficial if you set up Alert manager, How we can do?