Raspberry Pi IoT Server Tutorial: InfluxDB, MQTT, Grafana, Node-RED & Docker

[@nathanpendergrast6917]

Maybe I just hadn't looked hard enough before but I think you're the first person I've seen explain grafana this well.

[@familyplans3788]

This is ,by far, the best video I have seen for explaining an IOT server, thank you

[@frollard]

After what feels like 45000 tutorials, thank you - you finally gave me the insight needed to strip out unnecessary data and CHANGE the incoming data into a payload that influx was happy with. I now have a happy little chart of how bad the air is during the Canadian wildfire season.

[@LincsOracle]

Just a note, when setting up the MQTT server in Node-Red you can type in the name of the docker container as it is running in the same stack as Node-Red.

[@JoachimGrossmann-j6w]

Thank you for this excellent project & tutorial. It was my first step into the world of microcontrollers.

[@ika9]

Thank you! Your video is straightforward and to the point. Other videos were a complete waste of time. I'm glad I found your videos

[@nosafetyswitch9378]

Amazing guide! I watched it a couple of times and after a few hours I had my Grafana up and running, mind you I had never done anything MQTT before. For now I am just logging the room temp with an ESP32 and an LM35 temp sensor but I will be adding a lot of stuff soon and hopefully some solar production data in the future. Thanks a lot!

[@pedromorgan99]

Thanks

[@LearnEmbeddedSystems]

Thank you for the generous super thanks! I really appreciate your support :)

[@kylemfwelch]

This is a really excellent tutorial. Every step is clearly explained in detail. Just one issue: it is generally not advised to use a leading forward slash '/' for the root an MQTT topic. But that's a minor quibble

[@r4nd0mstuf]

After struggling for a whole day trying to get mosquitto to work on my raspberry pi, following your video had me up and running within half an hour. This is a fantastic tutorial, thank you so much for the help ❤

[@Bob-Fisher-6120]

Hey I also noticed that the "influxdb out" pink dragbox you used in the video at around 7:48 no longer seems to be in the storage tab, or in the left toolbox space at all. What happened to it?

[@SDNTechForum]

Amazing Video, cannot thank you enough. I tried it myself and here is feedback. The github code you shared is having issue with NodeRed, see error below - I managed to install other containers by skipping nodered and later installed nodered as systemd. It would be nice if you can make your github repo public in that way users can report issue and can pull the working code. ############ Start Stack: docker-compose up -d --remove-orphans WARNING: Some networks were defined but are not used by any service: nextcloud Creating network "iotstack_default" with driver "bridge" Building nodered ERROR: Cannot locate specified Dockerfile: Dockerfile

[@ceecee_22]

Really good tutorial. I wish I watched this before setting up my IOT server

[@srikumarc]

Fantastic Video. Very crisp and to the point. Helped me learn the whole stack at one go.

[@LearnEmbeddedSystems]

Great to hear!

[@srikumarc]

@@LearnEmbeddedSystems Is it possible to build this stack in the cloud? In other words can this container be ported to the cloud? A video on it would be helpful.

[@manasXP]

Great tutorial to get started with IoT stack in R-Pi4. I am going to use this in a CM4 and use node red + mosquito talking to IO module downstream. Docker surely makes things flexible and easy.

[@edinetgrunhed6000]

Thank you for this video, i learned about portainer,grafana and influxdb in same video. thank u very much

[@statters]

Great helpful video and website. One point, on the website it says to use 'docker-compose up' to check the status of the containers whereas you use 'docker-compose ps' in your video. I really appreciate the effort you are putting into your videos and supporting material.

[@eveb843]

Great Video, I am bumbling through this on my own system, I intend to use this to monitor my solar install, hopefully remotely, so much to learn so it may be sometime before it goes live. Thanks for posting ;)

[@yannkitson116]

Great tutorial... I am going to get the data from my netatmo devices and data from my "solar system" into this.

[@jimaustin3608]

(3:29) " fresh install of headless Raspbian OS ... (3:37) we have a video Linked In the cards above ... " Anyone know where this link is?

[@chrisrudy1969]

That was absolutely fantastic! Thank you so much!

[@paulhatfield498]

Very cool!! Thanks a heap for these awesome videos.

[@dang48]

Well done and I have a number of projects in mind that will use what was explained here.

[@richsadowsky8580]

Really great video with written instructions. Thanks.

[@RichardPaul93]

Thanks for this video, alongside the ESP32 IoT sensor node tutorial, both have helped me to quickly and easily get this connected! I'm looking for next steps - a graphical front end to show a user graphs from just selected nodes, buttons to perform actions, sending mqtt back the other way to the ESP32, etc. Any plans to continue videos on this IOT topic? Or anyone, any keywords you can point me at to research and learn appropriate components for this tech stack, such as a user front end, and user/device management?

[@eveb843]

Can I ask a question, how do you enable the USB serial ports in docker, I want to use Arduino nodes to collect and send data over serial, is there a file I can edit for the Node Red container?

[@AJB2K3]

Thanks for this guide, I have referenced and linked to it because it helped me build a server on an M5Stack CM4 Stack.

[@energyideas]

Looking into this for a mesh network of environmental sensors including ground level pollution from vehicles. Thanks.

[@navinarajoo7426]

Hello, this was a great tutorial! I got a little lost at the Nodered part. How do I know what is the Nodered Json interpreter that I should use?

[@Darkk6969]

Very cool video on the Pi. For Arduino the ESP8266 NodeMCU is a cheaper option as it already have Wifi built-in and better power management. ESP32 would work too. Both very small form factor.

[@michaeljohnson9758]

100% esp devices are better for 95% of iot projects and they are cheaper and smaller. Only problem is they are 3.3v devices so you might need a logic level shifter.

[@austinsandford]

I use wemos mini clones plus bme280s for very compact temp/hum/pres sensors

[@ShriyaSinhaBLC]

can even upload these data to an app and control the actuaters from it only?

[@HeleneLive]

Hi there, after running the linked IoT install script and script was processed, my Raspi boots up into blackscreen, no GUI available any longer. Luckily, I had a VNC server running, so I am able to connect via VNC viewer from my PC, although with a very poor resolution, that makes it nearly impossible to operate it this way. I have tried several reboots, with no success, obviously the script has "broken" my GUI, do you know how to make it available again ? I can access ssh via VNC. I'd like to avoid having to reformat my SD card again, Cheers mate !

[@josetuz]

Incredible! Thanks for that. From Argentina.

[@Bob-Fisher-6120]

Hey quick question, so when you get to the menu part and have to select nodered to install in the build list, do you have to download every single package that nodered has or can you just install the default ones shown?

[@KW-hy2bu]

Can I use esp32 instead od arduino with jolly module? Does it have a chance to connect with RPI?

[@mitchellboswell186]

Great video! Nice work. I would love to see the set up of the sensor node

[@sajadmohammadi3155]

Hello, I want to set the parameters of an industrial drive using a node through RS485 serial communication. In this project, I need to use the hex codes of an industrial drive and control the drive using the hex codes. Can you help me with this project with node-red-node-serialport?

[@markshaz8691]

Really well explained and informative video, subbed.

[@smaroukis]

I've been debugging my IOTstack install for a week now -- I found that I had a corrupted SD Card. Best video I've seen so far getting right to the point, explaining what **should** happen. It would be nice if it included some more gotchas - I've been battling with my v2 B+ which is pretty slow running the IOTstack menu.

[@LearnEmbeddedSystems]

Maybe I could put an FAQ or common problems and how to solve them in the written article? Would that be a good compromise?

[@stevebot]

Storage problem is my biggest concern with Pi. Consider one with onboard eMMC or a M.2 NVMe. There are a few options out there using the CM4. Seeed minirouter, Waveshare IO boards- base B had a RTC, or the DFRobot mini router to name a few. The latter two require finding a CM4, which everyone says is on back order until Dec 2023.

[@smaroukis]

@@LearnEmbeddedSystems that would be helpful, although I've just went the route of installing locally without docker containers and it is working

[@lpz3sn]

Running a database where data like temperatures etc. are constantly written with an SD card is only for people who enjoy reinstalling Raspis. USB-SSD/HDD is mandatory for this application.

[@olesik]

@@smaroukis I had the same problem, i have 5 corrupted sd cards on my desk now, I was about to try this videos method, I installed each item individually and the docker install script was killing sd cards

[@Nokless]

love the video !! can you make another video about the hardware setup plz ? and how the arduino sent data to raspberry ?

[@GiuliaLeali]

Hello! Great tutorial, thanks! However, when I build the docker-compose.yml.file containing Mosquitto, Influxdb, Grafana, Node-RED, and Portainer-CE and I type the command "docker-compose ps" I don't get any list of Mosquitto, Influxdb, and so on. I get the following error responses (see the attached picture). I've been trying to troubleshoot it and rewrite everything from the beginning, but the issue persists. I've tried to add some other commands found on the internet, but no luck yet. Let me know if you have any ideas on how I might solve this problem. Thanks :)

[@sy3002]

Which raspberry bi model is used here ?

[@shobhamaurya1886]

If I'm using influxdb on web browser instead of installing it to my pc, will it not work? Because when I'm connecting nodered with influxdb it's not binding with influxdb. I've used the host name for influxdb as my url and port as 8086.

[@patrickpuentelml4958]

I have a question. This thing (API?) called "IOTstack" was made using docker?

[@rafaelk2xt]

Well Done. Thank you. Do you know if can I do that on an EC2 AWS instance?

[@luqmanzainal3428]

Why do it shows “docker-compose : not found” when i wants to start stack? Help mee

[@radderek007]

You are inspiring. Thank you for sharing.

[@petermarton7083]

Hi! I did everithing, but the datas not arriving to Grafana from InfluxDB. On Pi I can see InfluxDB datas. What could be the problem?

[@monsefrouidi1261]

Did you find any solution?

[@contouraerials]

in theory this can be done on a headless pi zero from a pi because youre using the ip address to setup all the interfaces? (im a newb)

[@cernejr]

I am confused about Node-RED. Is it design-time-only tool? Or does it have to run also at runtime, after I deploy my solution?

[@sanjeevjoshi6441]

it worked fine for a few hours and then started returning this error ( rc=-2Attempting MQTT connection...failed ) what might be the issue

[@jesus_songs3005]

0.01 is this the circuit connection i need to make

[@adamk9559]

Was so hopeful - but hit a big problem. When i run the Start Stack command, it dies quickly at the Grafana step "ERROR: no matching manifest for linux/arm/v7 in the manifest list entries" (FYI - i'm installing onto Rpi 4B) I googled that error, and found multiple questions about it for other packages, but i really don't understand enough to know how to fix it here. I'm stuck. Anybody willing to lead me out of this road block?

[@BAXEN]

Has anyone been able to enable and enforce users for the mqtt container?

[@shivambandekar5329]

Can you do it with websocket and through public internet?

[@RudW0lf]

Based on my 3+y expirience InfluxDB wrong choice for time-series data. It can't be used on production setup. The better way - prometheus, or graphite with clickhouse backend.

[@YigalBZ]

Where do you discuss the static IP? Should this be set in the router?

[@toastedregret1601]

It should be set within your router, otherwise it will change the next time the raspberry pi power cycles. The video didn't mention that. Go to the IP address for your router, type the username and password (the very first time, it should be "admin" , "password"), find the raspberry pi in the list of devices on the router, view more info, and set "reserve" IP address.

[@MashiurRahman-ew8eq]

This is a very useful tutorial and a nice presentation!! Just to be clear regarding viewing the containers using Portainer, 1) do I need to have an account in Portainer? 2) Is there any limitation on device numbers under the free account?

[@jonamleonel]

if the RPi is running as a server, how from an outside windows computer could check the database and dashboards? or do I need to enter to RPi to check that?

[@toastedregret1601]

You can SSH into the raspberry pi similar to a virtual machine. There are tutorials online on how to do it. It's sometimes referred to as "headless mode". You can also take it a step further and access the raspberry pi from anywhere in the world by exposing its port on the router. That's a huge security risk though - you can also alternatively use an application like VNCViewer, set the raspberry pi address as static so it doesn't change on startup, and access the raspberry pi via VNCViewer from ideally anywhere.

[@homelessgee957]

Do you have any suggestion to decrease power usage to change it to make a reading every 1 mins instead of few seconds. And using only solar energy from a panel? To make sure its fully autonomous

[@toastedregret1601]

You could look into lower-power mode on the device you are using, and use a watchdog timer or some other peripheral on the microcontroller to power it back on after X time to take another measurement.

[@pllagunos]

This is a really amazing video. Thank you. Any chance of you doing a tutorial on how to deploy Grafana Cloud connected to the local RPi Server? That way one can monitor devices away from home and even have alerting through Grafana IRM's mobile app.

[@Scobbo]

So one topic is for each arduino with the sensor suite, so each room has a different topic? or is it a bunch of sensors sending data to one arduino and it is transmitting different topics per room?

[@toastedregret1601]

Each arduino is an individual sensor node in the video. There are cheaper microcontroller alternatives with more IoT features (timers, power modes, WiFi, BLE), such as the ESP32. Arduino in this case was used as an example.

[@vedantjadhav4525]

after configuring everything the datatbase is not showing any measurements

[@guillaumecuadrado8096]

Hi ! I got the same issue ! Did you managed to fix it ? thx a lot !

[@vedantjadhav4525]

@@guillaumecuadrado8096 yes just experiment by replacing sensor with sensors and vice versa in few area he has put different spelling

[@galuyoogank3705]

this video is supposing all devices are connected to the same network, right? Is there a way to implement same solution you provided, but with devices that are around 20-40 metres away from each other. either by somehow giving them same network or by working it out without same network.

[@redikaardikusuma2296]

I think VPS digital ocean is the solution

[@michaelandersen1161]

Great video....when I try to install thsi with a static IP adress set I can not connect to the server. When must I set the static IP adress?

[@momaloku9896]

Is there any code for Raspberry Pi Pico W? Or has anyone tried this setup with that board?

[@javierfernandezmarcos120]

how do you configure MOSQUITTO? I selected the IP of my RPI3 and I am not receiving anything :(

[@Lucas-xu5oq]

Well this video was exepsional congrats! It worked fine for me, however for some reason I can't add the libraries of the functions on the pallet in node-red.

[@trevorwatson5041]

Mine did not work as when i want to start the stack it returns an error that three of the apps are obsolete. Just wondering if you had an issue like this or any way to resolve

[@HenrikBoensvang]

I was wondering. Could anyone point me in a direction where I'm using a microbit instead of the Arduino? Why.. I don't have an arduno :(

[@willembos01]

Very good tutorial!! Thank you I will experiment with this.

[@patrickpuentelml4958]

Almost every single thing is very good explained in this video an exception of HOW CAN I MAKE THE INTERFACE COMMUNICATION BETWEEN MY ESP32 (WHICH IS READING THE DATA SENSOR) AND NODERED?

[@JustinMannI_T_I]

Greetings, I am really stuck. I can't seem to access the mosquitto.conf file for the container, which i must access to "allow_anonymous true". When I try to start a console from within portainer I get "Error Unable to retrieve image details". I can't seem to figure out how else to edit the mosquitto.conf file?!

[@thorstenfricke3257]

have you ever used iobroker?

[@alsamaraee2]

Hi, Thank you for this professional tutorial. I followed it and my ESP32 client can connect to the MQTT server jus if both the Raspberry pi and ESP32 on same wifi network. What am I missing?

[@oskarbrenner13]

I think MQTT is run upon a network, such as Wifi, bluetooth e.t.c.

[@john_hind]

I've been trying to get a similar setup but using CasaOS instead of IOTStack (but resorting to IOTStack for hints about how to set up Mosquitto and Influxdb which are not in the CasaOS app store). This gives a nice web front end to the various web apps and easy remote file access. Was wondering if anyone knew of a web app for Mosquitto, something that would allow web page monitoring of message traffic (similar to MQTTX, but web) and maybe some configuration GUI avoiding all the command line and config file editing jugitsu, similar to what Portainer does for Docker. Likewise for InfluxDB. I am very confused about what URL to use for one container from within another. What are the (different) URLs listed in Portainer for each container for? I found using the specific URL for the Mosquitto container worked in some cases where the URL of the Pi did not.

[@toastedregret1601]

Maybe wireshark to look at the datapackets being sent over the network? I know it's useful for UDP.

[@akshatmodi4843]

Hey LES! Superb video thank you so much for the information. Just had one question, I have certain scenario's where people switch of the Raspberry Pi at the end of the day's work. So when they restart will all of the tools namely, Mosquitto, Influx-DB, Node-Red and Graphana or essentially the docker container reboot as well? Thanks again for the fantastic video.

[@javieralonso4054]

Hi. This is very good work However, I'm trying to develop this with my raspberry pi and an arduino MKR. It happens that the WiFi library has its init method as private method, so no way you can start the WiFi in AP+STA mode (at least, I've not been able ot do it). Is there any work around. I think this is the cause the sensor node only gets one data through the node red workflow.

[@HolgerEich-k9h]

Hi Javier, I’m interested in using the mkr, too. Have you received any feedback or found a solution? Best, Holger

[@YigalBZ]

Whn isntalling the stack, an error appears: not enough lines to render. Any idea how to avoid it?

[@YigalBZ]

Solved it by maximizing the window size

[@SobhitPanda]

Absolutely awesome content 😊 + Your voice 🌟

[@LearnEmbeddedSystems]

Thank you!

[@abhishekpaul97]

Hi, extremely helpful video! Can one use the sensor data to control GPIO of another raspberry pi using MQTT?

[@feedmemusictv]

Thank you. Nice; quick and succinct. One comment.Why skip over Portainer when - and this is from limited knowledge - I think you could have set everything up from within Portainer

[@jericocura1546]

does mosquitto still work?

[@fra8156]

very useful video, I did this Iot Server, but with DHT22 temperature sensor with ESP32. Did anyone have the problem that at 7.30 am and pm for some reason grafana stop saving data? Any suggestion?

[@austinsandford]

Great tutorial, would be useful if Grafana could easily plot multiple values on one graph so that many sensors could be viewed easily

[@janhatton4405]

Hi, yes it can do this you just add a second query panel when you are setting up the graph (+ Query button) and it outputs to the same graph

[@austinsandford]

@@janhatton4405 Thank you

[@PurnimaMishra-g7o]

Absolutely brilliant.

[@237311]

Great tutorial. I have so many Pi 3s lying around, and wanna know if I can use them on this project? Now Im interested to see some videos on how tos with the sensor nodes.

[@LearnEmbeddedSystems]

Pi 3 should work just fine! ESP32 sensor Node video hopefully coming this weekend!

[@makmakmp]

That's an excellent tutorial !! Congratulations !! I would like to ask in case we have many sensors in /home/sensors. How can we get the unique id of each sensor and show in grafana? I've implemented your project with wemos D1 mini + DHT22 sensor + LCD 16x2 (parallel connection) and raspberry pi 4 (8Gb) as the server. The point is that viewing grafana i only see that data comes from sensor_data which is the db name from influxdb.

[@sanketss84]

Superb video ❤

[@vitusyu9583]

I am a novice in IoT, and appreciate if you could explain how to do the sensor board part. Thanks!

[@toastedregret1601]

The video uses an arduino as the microcontroller unit (MCU), but if you don't want the hassle of connecting a WiFi module to it and managing the communication between the two modules, you can get a microcontroller with wireless capability (bluetooth, WiFi) embedded directly into it. I'd reccomend using an ESP32.

[@thomasgrassmann6724]

Great video! Thanks!

[@duckyblenderold]

Very informative video, thanks!

[@YigalBZ]

After installing the stack I can see all components up as shown 06:18, yet when trying to run the browser with port 9000,it says: "New Portainer installation Your Portainer instance timed out for security purposes. To re-enable your Portainer instance, you will need to restart Portainer.". Restart the RPI didn't solve it. Any idea?

[@YigalBZ]

Solved with docker stop portainer-ce, then docker start portainer-ce

[@AndersJackson]

Great video. But as I understand it, MQTT messages should not have a starting "/".

[@KeesFluitman]

I kinda miss the hardware side info?

[@nirojkumarsahoo]

Very informative. Definitely going to try this.

[@LearnEmbeddedSystems]

Let us know how you get on!

[@ChenLaw88]

Thank you! Very nice, well explained and easy to follow, bookmarking and saving your website tut too. I've got a couple renewable energy projects that I'm designing, and this should work perfectly for some monitoring hardware (tachometers on wind turbines and voltage readers on solar arrays). Also thinking about a few other metrics around the house I could definitely appreciate collecting :D.

[@najim5mmmmm]

Hi, thx for the nice tutorial but as some people said earlier, I think there is an issue with th IOTStack and Node-red. Indeed while installing the stack the installation gets stuck and Node-red is not installed. I found out the reason was because I have configured the Raspberry IP as Static ( like shown in your previous tutorial :) ) It seems that during installation an IP has to be assigned to the containers and without DHCP it gets blocked... Can it be?

[@YouKnowWho4828]

This exact thing happened to me, I also have a static IP set in /etc/dhcpcd.conf. Disabling the static IP resolved it. Nice find!

[@slambergamer91]

can you access grafana using phone for viewing ?

[@LearnEmbeddedSystems]

Yes! Just access the IP address on your mobile. If you want to access it remotely (not on local network) then you will have to vpn in.

[@Marzanitrate]

help i got stucked in this situation. IOTstack Installation Enter in the sudo password when prompted, to install dependencies Reading package lists... Done Building dependency tree Reading state information... Done git is already the newest version (1:2.20.1-2+deb10u7). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. fatal: destination path 'IOTstack' already exists and is not an empty directory. IOTstack cloned Python virtualenv found. Python Version: 'Python 3.7.3'. Python and virtualenv is up to date. Error getting docker version. Error when running docker command. Check that docker is installed correctly. Docker version 23.0.1 >= 18.2.0. Docker is good to go. Setting up environment: pi@raspberrypi:~ $

[@spillagonner]

Please note: In the video, you state that you can check what containers are running with the command 'docker-compose ps' while on the related web page you show the command as 'docker-compose up'

[@ivanronda9583]

docker-compose ps shows you status of docker containers, docker-compose up execute the docker-compose.yml file(that is something like a config dile for docker) you also use this command only when starting docker