Best Server Monitoring with Prometheus and Grafana using Node Exporter and cAdvisor

A complete walkthrough of completely free and open source solutions to monitor your Linux, Windows, and Docker servers. In the view we see how we can setup and configure Prometheus, Node Exporter, Grafana, and cAdvisor to monitor servers in our environment and how you can easily configure these.
Clone down the examples in this video from the following Git repository:
github.com/brandonleegit/Open...
Subscribe to the channel: / @virtualizationhowto
My blog:
www.virtualizationhowto.com
_____________________________________________________
Social Media:
/ vspinmaster
LinkedIn:
/ brandon-lee-vht
Github:
github.com/brandonleegit
Introduction to Prometheus, Node Exporter, Grafana, and cAdvisor - 0:00
Looking at how everything fits together - 1:15
Node Exporter details and written for Linux - 2:15
Windows Exporter to expose metrics for monitoring Windows machines - 2:28
cAdvisor for monitoring Docker containers - 2:56
Looking at the Linux host serving as the Docker container host - 3:26
Installing Docker Compose - 3:51
Verifying Docker Compose installation - 4:12
Looking at our Linux user used to run Docker on the Docker host - 4:27
Creating the local folder structure for the local volume mounts - 5:05
Using a crude bash script to create the folder structure - 5:23
Running the command to create the folders - 6:29
Overview of installing the monitoring stack with Docker Compose - 7:08
Editing the Docker Compose file for creating the monitoring stack - 8:03
Stepping through the Docker Compose configuration - 8:18
Talking through permissions and adding the User ID to avoid permissions issues with Prometheus - 9:22
Looking at cAdvisor and odd ports used - 10:47
Redis dependency 11:10
Saving the Docker Compose YAML file - 11:34
Running the Docker Compose UP command - 11:53
The provisioning of the containers finishes, checking everything to see if it is up and running - 12:18
Talking through troubleshooting if a container is crashing - 12:58
Containers are healthy and ready to move forward - 13:13
Talking about configuring Prometheus and getting metrics into Grafana - 13:23
Beginning the configuration of Prometheus - 13:43
Adding configuration to the Prometheus YAML file - 14:18
Stop and start Prometheus container to pick up the new YAML file - 14:43
Browsing out to port 3000 on Docker host to login to Grafana - 15:08
Change your password - 15:18
Navigate to settings and adding a data source - 15:30
Populating the address of our Prometheus installation - 15:48
Save and test the new data source - 16:06
Looking at the Explore dashboard icon - 16:17
Importing a community dashboard - 16:40
Getting the JSON file or ID from the Grafana community dashboards site - 17:00
Downloading the JSON or copy ID to clipboard - 17:30
Paste in the ID of the community dashboard - 17:42
Immediately you see the new dashboard imported to view metrics - 18:15
Looking at Windows Exporter to import metrics from a Windows host - 18:50
Download the Windows Exporter file and copy out to Windows host - 19:16
Running the Exporter installation and viewing Windows Exporter service - 19:30
Adding the Windows Exporter host to our Prometheus configuration - 19:50
Restarting Prometheus to pick up the new Windows host - 20:23
Download a Windows Exporter community dashboard - 20:37
Viewing the new Windows Exporter dashboard and information - 21:18
Talking about adding Docker to monitoring with Prmetheus and Grafana - 21:47
Looking at the Prometheus configuration and the cAdvisor configuration we have in place - 22:04
Concluding and wrapping up thoughts on Prometheus, Node Exporter, Grafana, and cAdvisor - 22:53
A written walkthrough of Prometheus Node Exporter monitoring:
www.virtualizationhowto.com/2...
Best docker containers for Synology NAS:
www.virtualizationhowto.com/2...

[@romayojr]

just spun this up as an lxc docker container on my proxmox cluster and only took less than 30 mins without any issues - amazing tutorial, brandon. cheers!

[@guapoforastero2668]

First off all thanks for the excellent vidéo it has helped me in my project.

[@damadorpl]

a little hint with today docker 23 update :) docker-compose up -d become docker compose up -d - wthout dash in the middle :)

[@guidoricioppo7759]

Hello, great video! From what I understand Redis is not actual dependency. It is used on the prometheus guide just as "reference container" for examples

[@VirtualizationHowto]

Thank you for your comment! Good point!

[@itsmendyou]

Thank you. It's very helpful.

[@ThrottleScotty]

Excellent tutorial. Thanks for this! I'm just getting started with Prometheus and Grafana. Have two questions: 1) Have you seen any dashboards that aggregate Linux, Windows and Docker monitoring into a single (perhaps very simplified) view? 2) Do you have recommendations for alerting? Meaning, if a docker container is reporting as unhealthy, or a specific Linux OS metric is reporting above/below a set threshold, an alert is sent via email/sms to admins...

[@Synthel98]

Great tutorial! Do you have some guide to read better logs of all containers? Currently I'm using Dozzle, but I saw some solutions like ELK (Elastic Search, Logstash and Kibana) , any chance to you bring some tutorial to implement in docker-compose?

[@guapoforastero2668]

I have a question what would the configuration of the exporters and the Prometheus server be like if the servers were AT the company's office and the Prometheus server was AT home. Basically a remote monitoring. Thanks

[@NetBandit70]

Heavy on the configuration. You'd think that a monitoring agent would be able to push metrics out instead of listening. So now you have to install the agent (Windows Exporter), configure the Windows firewall, and setup Prometheus to pull metrics. Anyway, a great video even if the configuration is cumbersome.

[@filipfedorisin3457]

Hello, i am unable to hit node exporter and cadvisor metrics endpoints via prometheus. It gives me context deadline exceeded error. However when i try to curl it works like charm. Can you please help me with this?

[@md.ishfakurrahman8457]

How do you decided ip address for targets? There is docker ip address too!

[@bayuwiratmo2820]

how to add target if I have 5 VM running on premise server? Just newbie

[@victormacedo4080]

Great video! Does anyone knows how much CPU and memory this monitoring stack uses?

[@aoczon]

Not sure if this has been asked already. Can you add how also how install the snmp exporter. Thanks

[@mathewcampisi7594]

How do I add other servers to prometheus? Like 2 linode and a digital ocean server? I'm running prometheus locally, so would i need to create cloudflare tunnels???? Also can you make a video on running grafana and prometheus on trunas?

[@janupriyo2096]

so easy turotial, Good job thanks bro

[@VirtualizationHowto]

Janu thanks for the comment! So glad it was helpful.

[@janupriyo2096]

@@VirtualizationHowto you are welcome bro

[@andreaquentino1068]

nice!!! Very handy!!!

[@VirtualizationHowto]

Thank you Andrea!

[@squalazzo]

i beg you pardon in case you already explained this in the video (i'm still at the beginning), or you can maybe make a video on this in the future, please: having BOTH metrics AND logs combined in a grafana dashboard, so to have a complete monitoring solutions, scrolling both "targets" in sync/parallel, to have a full view on what, how and when something happened... thanks!

[@squalazzo]

ok, saw full video, probably what i asked is related more to kubernetes and less to docker, really... but, in case, a video covering this "extended" setup would be much appreciated :D

[@MrInvalidnick]

Thank you for share. Greetings

[@VirtualizationHowto]

Thank you for your comment! Take care.

[@leolomonosov4021]

Hello! Perfect tutorial. The only problem for me, where do you get the IP for grafana? Should I go by localhost or I need to find grafana's IP?

[@opozicijanajednommestu5682]

ifconfig

[@RolandTutorials]

i love this, thanks

[@VirtualizationHowto]

Thank you too!

[@vasiliynet3425]

Thanks! Looks more pretty than Zabbix.

[@VirtualizationHowto]

Vasilly, I definitely prefer PRTG for its capabilities, look, and feel...

[@typingaddict1319]

It doesnt seem like grafana can see the nodes i have from cAdvsior. My Docker container dashboard has no data. What steps can I take to troubleshoot this?

[@witblack]

i have this problem too. Did you find any answers for fix this issue? Can anybody help us?

[@886609769]

need some assistance: Prometheus is up and working, I have node exporters loaded on few server. Prometheus is able to identify them and scrape them. Installed Grafana and connected to Prometheus but Grafana is not able to show any data in dashboard any recommendations?

[@VirtualizationHowto]

Hello and thanks for the comment! Does grafana see the prometheus data source when you add it in settings?

[@daiben2004]

nice, But how to monitor VMware ESXI and True Nas Core Storage?

[@VirtualizationHowto]

@daiben2004 Thank you for your comment! Sign up on the forums and I can give more personalized help here: www.virtualizationhowto.com/community. Thank you again.

[@shamshershaikh17]

Can we create a metrics to monitor service status like ntpd, chronyd, auditd in Prometheus

[@VirtualizationHowto]

@shamshershaikh17 thank you for the comment! Sign up for the forums here: www.virtualizationhowto.com/community and let's discuss it in more detail there.

[@sadesguy1152]

mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied

[@MarkConstable]

Nope, not interested in a docker based solution... moving on.

[@majorgear1021]

Aww. Come on. Try it.

[@rainbowforest09]

I am having an issue with prometheus. When I try docker-compose up -d there is one error that occurs. Cannot start service prometheus: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/root/monitorsvr/prometheus/prometheus.yml" to rootfs at "/etc/prometheus/prometheus.yml": mount /root/monitorsvr/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? How to solve this?

[@VirtualizationHowto]

erefem make sure you created a directory and not a file in your Linux environment. Also, check out permissions. Let me know if this helps.

[@rainbowforest09]

@@VirtualizationHowto I follow every step carefully but have had no success. I also add user to docker group. Is there something I am missing here?

[@fortypanda]

@@rainbowforest09 1. Switch your user to root and run everything from there. 2. Check the user id (id -u) then change it to match accordingly in the docker-compose.yml file as in the video. Then proceed with the steps as he has done. It will work.

[@TerryMcKnight1]

I have exactly the same issue as erefem. lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.10 Release: 22.10 Codename: kinetic /promgrafnode$ sudo docker compose up -d [+] Running 4/5 ⠿ Container redis Started 1.2s ⠿ Container cadvisor Started 2.4s ⠿ Container node-exporter Running 0.0s ⠿ Container grafana Started 0.0s ⠼ Container prometheus Starting 2.4s Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/root/promgrafnode/prometheus/prometheus.yml" to rootfs at "/etc/prometheus/prometheus.yml": mount /root/promgrafnode/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

[@khisanthxanadu2192]

I get the same error