#255

[@HendraKusumahiot]

"We like programming, but we love result" Quote of the day. Thank you sir

[@AndreasSpiess]

You are welcome!

[@bigheat86]

Even I can confirm, and I'm a developer.

[@lightpixeldotnet]

this :]

[@richard_wenner]

Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).

[@AlenTuzlak]

I agree 100%. Andreas is amazing!! Thank you so much Sir for sharing knowledge with everyone and great commitment on making amazing videos !!

[@AndreasSpiess]

We are slow here. That is probably why our time runs a little slower... I hope the content still helps you a little for your pollution project!

[@richard_wenner]

@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.

[@AndreasSpiess]

Security is not my specialty, fortunately ;-)

[@aux789]

Amazing! I had to stop the video on two occasions I was so overwhelmed by the sheer power of this suite of FREE software! Thanks Andreas!

[@AndreasSpiess]

That is the power of KZbin. You can use it at your pace ;-)

[@voltlog]

This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.

[@AndreasSpiess]

I think it is not so easy if you have to do everything from the ground. So we both were happy we were able to stand on the shoulders of others ;-)

[@chuxxsss]

I did find another script from another person with red-node. It is handy for solar and batteries. I hated manually install parts of Peters script the .npm title. But Peter has done a great Job with this script with Mr Sharky. I love it. Peter is a great person thank goodness we have him around. The graphics I think was made by him too.

[@AndreasSpiess]

With his new version, you should not need any manual install...

[@hsmptg]

Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!

[@AndreasSpiess]

I mentioned the compression, but not in detail. I do not plan a video on that because it is very special. Maybe once part of another one. Who knows?

[@PeterScargill]

Nice to see you're making good use of "the script" Andreass.

[@AndreasSpiess]

I always like to stand on the shoulders of big men ;-) Makes me more productive! Like that, everybody can profit.

[@PeterScargill]

@@AndreasSpiess Hah. I do that all the time. My script enables software like Node-Red, written by people cleverer than me!

[@deangreenhough3479]

Thank you both for your wonderful work. I have a beautiful image on my Pi and it’s all working 😁🏴󠁧󠁢󠁥󠁮󠁧󠁿

[@AndreasSpiess]

The timestamps of the example data are from 2019-02-06. Please choose an appropriate date range in Grafana if you want to display them

[@springwoodcottage4248]

Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!

[@AndreasSpiess]

You are welcome. And I agree that retention strategies are not easy to define these days with very cheap disk space...

[@opsahle2]

Extremely usefull video - I am very happy to be one of your Patrion-supporters! Keep it coming, Andreas!

[@AndreasSpiess]

Thank you for your support! It is always good to read that my videos are useful...

[@rgrf]

Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.

[@AndreasSpiess]

Maybe you also watch my videos about IOTstack...

[@rgrf]

@@AndreasSpiess I did Andreas and for the first time doing useful things with this stack. More thanks. My stuff is at rogerfrost.com. Best wishes.

[@hoekbrwr]

This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!

[@AndreasSpiess]

I transferred historical data exporting to CSV and creating the needed input strings for influxDB with Excel.

[@yura979]

I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O

[@AndreasSpiess]

Maybe you can save some time now ;-)

[@Argosh]

I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!

[@AndreasSpiess]

I hope you will be successful in transferring the data from your source into InfluxDB.

[@lucapesce3335]

Cool, I use InfluxDB for my home, but never understood the retention policy, thank you. I have three point about your setup: - InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination - This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this) - All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover

[@AndreasSpiess]

1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot. 2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.

[@noealves8400]

Mr Spiess, my deepest gratitude for your work and for sharing great content in a video like this. I became your Patreon today to support your work!

[@AndreasSpiess]

Thank you for supporting the channel. I appreciate it!

[@mrtarantl]

Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...) Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) ) Greetings from Austria

[@AndreasSpiess]

Thank you! Jokes are not easy in an international environment. Glad to read that you like them. I like music, but in videos like that, I hate it myself. This is why I do not use it...

[@ursmii_]

I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video

[@AndreasSpiess]

Very fast! Did you try the docker Installation?

[@ursmii_]

no I did not. I'll try to create a surveillance for 24/7 running PCs (if message does not arrive periodically - alert)@@AndreasSpiess

[@AndreasSpiess]

Interesting project! We created a whole project with lots of useful containers (IOTstack)...

[@santorcuato]

I have no words Andreas. Astonishing. Thanks.

[@AndreasSpiess]

You are welcome!

[@Alby_VA]

Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,

[@AndreasSpiess]

I also like Grafana...

[@Alby_VA]

@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)

[@mikethezipper]

You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.

[@AndreasSpiess]

Glad to read that!

[@LucasGuillemette]

I'm a programming student and I was this kind of data retention policy saves a lot of work. Glad I don't have to reinvent the wheel.

[@AndreasSpiess]

We live in a good time with all this open source software...

[@UpLateGeek]

I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.

[@AndreasSpiess]

MongoDB seems to use quite a lot of disk space because it is made for other purposes. There is a comparison on the influxDB page...

[@UpLateGeek]

@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.

[@sausagehider]

Such a coincidence. I was up until 1:30 am this morning looking at almost the exact same thing. Excellent content, I will give it a go today. Thanks. Martin.

[@AndreasSpiess]

Good luck. I hope it will work for you!

[@MirkWoot]

Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.

[@AndreasSpiess]

I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)

[@steveb7600]

You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.

[@portfedh]

Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.

[@AndreasSpiess]

Excellent!

[@brunosolothurnmann9205]

Thank you - I installed Grafana and InfluxDB on my existing Raspberry Pi 3+ with an already installed Node-Red and Mosquitto. For the installation of Grafana and InfluxDB I found some instructions on the internet. I use Node-Red and Mosquitto a lot for my home automation and used until now SQLite or flat files. Your Video helped me to setup it - and it runs fine. Now I'm going to adapt my home automation to the new data sources.

[@AndreasSpiess]

Now you can watch the newer version (dockerized) of this setup.

[@YouGenom]

Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)

[@AndreasSpiess]

So far I always used Node-Red and do not know Home Assistant.

[@YouGenom]

@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.

[@kokoscom]

Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)

[@kokoscom]

it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)

[@AndreasSpiess]

25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...

[@followthetrawler]

Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun. Thanks for another great video and the idea...

[@AndreasSpiess]

There are a few other videos about weather station hacking and Pi and Docker (IOTstack)

[@thorsten9211]

Thank You! I love how well structured your videos are. Suits my taste as german engineer ;)

[@AndreasSpiess]

Thank you! Maybe you also watch the one about Docker. It simplifies the installation of those programs

[@thorsten9211]

@@AndreasSpiess Yes i did and everything is working nicely together. :)

[@fpaolo63]

Hi Andreas, just to share. I created a "home heating system monitor" based on some arduino-nano, reading NTCs/DS1820, and raspberry interconnected by field-bus based on modbus (RS485). Raspberry is running a python script to collect data and store it on influxbd, Grafana is used to visualize data and correlate info. GOAL: integrate solar PV, heat pump system and Gas heat generator, control floor heating and try to be more green. ... will see...

[@AndreasSpiess]

It sounds like an interesting project. Maybe you have also planned to add something like hass.io? And maybe you also watch the video about IOTstack (if not already done)

[@fpaolo63]

... Yes I am looking deep in Hass.io ... but ... before go in Strong IOT every one should read 'Abusing the Internet of Things'. For now l prefer to stay with a well lock down reporting tools.

[@laveur]

Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.

[@AndreasSpiess]

At least you would have great support for your project. Not as we "simple people" ;-)

[@px1690]

Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.

[@AndreasSpiess]

Maybe I have to look at hass.io once...

[@ajaybnl]

Everything new. I had watched twice to understand. I want this setup too for my home sensors. Thank you dear.

[@AndreasSpiess]

You are welcome!

[@ajaybnl]

Im now successfully using grafana influxdb with mosquitto (EspMqtt) using node red on my OrangePi Lite.

[@klogaraa]

Amzaing effort from you and Pete's , keep the videos coming Andreas 👍

[@AndreasSpiess]

Thank you!

[@MrAtarifreak]

Thank you for your great video. These are the projects that I can learn from a lot....

[@AndreasSpiess]

You are welcome!

[@regenerativelife991]

thanks a lot for this research and experimentation. we will definitely use this for our farm.

[@AndreasSpiess]

I hope your application will add some value :-)

[@asiw]

This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on KZbin. Your viewers might find the comparison interesting.

[@AndreasSpiess]

SQL is always a possibility. It is stable and does its job.

[@squalazzo]

@Andreas issues are now solved, as you know by latest comments on Pete's blog... i'm MrShark :)

[@AndreasSpiess]

Thanks! The video was already out then. I pin your comment that everyone can read it.

[@squalazzo]

@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine

[@squalazzo]

@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)

[@MrJanulis]

@@squalazzo Influxdb node still requires manual installation - tried just a couple of hours ago

[@realsushi_official1116]

Very interesting and easy to jump in. Actually I was using a Python Jupiter Notebook for Receiving, parsing, sending payload to the MQTT broker. This solution is great, especially for future maintenance

[@AndreasSpiess]

I never thought about using Jupiter Notebook for something like that. Now I have to look into it. Tanks for the info.

[@familyplans3788]

just putting up xmas lights but as soon as that is done i will be following this video and cant wait to use Mr. Scargills 'script'

[@AndreasSpiess]

Maybe you watch the „ultimate pi server“ video first.

[@familyplans3788]

@@AndreasSpiess LOL i was just wondering that !! good job the mrs LOVES your accent so i get away with watching geeky videos

[@alanjrobertson]

Really helpful explanation of the benefits of time series databases for IoT, thank you 👍👍

[@AndreasSpiess]

Glad it was helpful!

[@MitchellWongHo]

Excellent demonstration of how little you need to collect and visualise data. I have a spare PINE64 which I'd like to try using these building blocks

[@AndreasSpiess]

You have to check which operating systems the script supports if you want to use it.

[@veljac99]

Nice video Andreas! In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before). Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting. I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon. Point is that any additional knowledge nicely fits in picture. Constant evolution...

[@AndreasSpiess]

Other commenters also mentioned that it is easy with Home assistant. Maybe I have to try it one...

[@ThWischi]

Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas

[@AndreasSpiess]

Thank you for your kind words!

[@einars899]

Interesting and useful video! It seems we are heading back to remote sensing? I like that.

[@AndreasSpiess]

Sensing and acting based on these data is IOT for me ;-)

[@wilman_studio]

Awesome guide to getting Grafana up and running, thank you :)

[@AndreasSpiess]

You are welcome!

[@princebanini]

this was an eye opener for me. always a good time learning from you

[@AndreasSpiess]

Glad to hear it!

[@wassfila]

I love it, now we really talk IoT. One more step Andreas, automated MQTT to influx converter. Let me know if you'd like to have the links. In my porject (google "home smart mesh"), I have such a python converter. It subscribes to a certain MQTT topic pattern, and posts jsons as is into influxdb. Everyone is already very familiar with MQTT directly produced from ESP and others, with an automated converter, you would not need to edit a custom translator for every sensor you add. Yeah and I forgot something, once MQTT is plugged to influxdb and grafana, you enpower it with something like zigbee2mqtt and have awesome cheap market products on you dashboard !

[@AndreasSpiess]

I would be interested in the links. Did you do a tutorial? So far I always thought that ZigBee is quite expensive for makers and not well accepted in our community. Maybe I am wrong...

[@DavidL3430]

Hello, I spent a quick 30 min digging into your “home smart mesh” with the objective of understanding just the mqtt to influx converter. I feel close to getting it but honestly got a little lost in trying to peel the mesh application out of it. In my case I have at least a dozen esp and other devices all speaking to mqtt over Wi-Fi to node red then over to RRD databases. I want to explore updating this to mqtt to influx to grafana directly. Examples: /sonoff/1/temperature/ or /base/55/humidity/ or /1wire/“serialnumber”/“englishID”/temperature/ Do I configure all these in JSON or just create rules for each generic example? And Bravo to Peter and team for that script which I have used several times on Pi and other Linux installs!!

[@RalfVogler]

Haven't tried it yet, but there's a Telegraf plugin for MQTT to Influx: github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer

[@wassfila]

@@AndreasSpiess for MQTT to influx with python please find a direct link to the github influx python directory here github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx I do not have a special tutorial for the MQTT to influx, but to be honest I struggle to make a documentation or tutorials as clear as you do, you are better at that Andreas :) Nevertheless I spent effort to document the whole "Home Smart Mesh" which has a lot of custom RF in it, and it is not easy to isolate the raspberry pi scripts from the rest. I tried to document on Hackaday hackaday.io/project/20388-home-smart-mesh and on Github which is more focused on the nRF52 Hardware github.com/nRFMesh/nRF52_Mesh When it comes to Zigbee, I can confidently say that it became cheap enough now with all sort of ready sensors ~< 10 € from Ali, and an official nRF52840-Dongle < 10€ from Mouser. Now I would agree with you if you say nRF52 Thread or Zigbee is not well supported by Arduino, but Andreas, we have to make it happen, by selecting best HW and pushing it to be come easy and popular. I started wrapping the NRF SDK official from Nordic around a c++ API with classes that make it Arduino like, my latest progress on that here github.com/Roblibs/rover_firmware/blob/master/applications/03_sensored_bldc/main.cpp but far from being a real arduino core. The nRF52840 is I believe a real challenger for the ESP32, as the nRF52 has native USB, an Automated Programmable Peripheral Interconnect, and with its custom radio I achieve latency ~ 1 ms. I'll be glad to share any required further info and details.

[@wassfila]

@@DavidL3430 please find a direct link to the python script that converts MQTT to influx, it should be possible to take that directory away from all the rest github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx But I admit, it is not packaged to be used without Python knowledge, as you notice, even if I have the main config in the config.json, a lot of the logic is coded in the def mqtt_on_message, for example the number of words in the topic e.g. mesh/node5/temperature and also I force some types to be float or int in lines 36 to 43. But on the good side, I even prepared scripts for backup and export, and the note that the config can be different depending on the hstname so that you have many raspberry pi and save all config in the same directory with names config_rasphostname1.json, config_rasphostname2.json, ... I will gladly provide you any further help if you want.

[@jab5569]

Nice work Andreas! Thank you for putting this together, very informative.

[@AndreasSpiess]

My pleasure!

[@manuelorellanahinojosa7356]

Your channel is pure gold!! Greetings from Chile!

[@AndreasSpiess]

Thank you very much!

[@athanasiosstefanopoulos5211]

What another great video, thank you Andreas! Your video inspired me for changing my implementation on how to use historical data.

[@AndreasSpiess]

I hope you will be successful!

[@athanasiosstefanopoulos5211]

@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana. And all these thanks to you again :)

[@realbss1]

Hi Andreas, relational databases do not necessarily have to be setup with fields like "Temp, Pressure, ...", you can as well do an different approach and use a dedicated field to indicate the type of the value. Something like "Timestamp, Type, Value", that way the only restriction is the type of value you want to store ( integer, float, string, ... ). If you want to get completely rid of that you should have a look at elasticsearch, it's a database system that is highly scalable and very dynamic. You can easily scale it up from one DB node to hundreds in seconds, simply by adding additional nodes and the same way back (in reality there is a little downside in downsizing - let me know if you are interested). Elasticsearch does NOT stick on types, or even database table schemas. It simply stores what you send - if there is no - for example Temperature - it just creates it dynamically for you. Additionally the values are the same flexible - no need to preselect a type like float or a length, it's just dynamically created on the fly. It's even able to store >100k Messages/s depending on the machine it's running on. In reality I had soe time ago a DB running where I was feeding event log data from 9 Active Directory domain controllers and keeping ~1 year of history of that data. Peaks where around 500k messages/s and around 5-7 million entries a day on a 6 year old reused hardware. Basically a perfect match for MQTT where we do not have strict type/name rules as well.

[@AndreasSpiess]

You are right. You can store these primitive sensor values in every database without any problems. AFAIK Elasticsearch is considered to be a "NoSQL" database. They are very good at certain things like you describe: Unstructured data and also full-text search. As I said, the strength of time-series databases is not the storage side but on the retention as well as the extraction of time-based data. I do not want to focus on database concepts on this channel because there are people which are much more capable than me in this matter.

[@RalfVogler]

It's a search engine, why should you use it as a database? How do you efficiently get the average of a field (or even better over a time period)? Types and structure shouldn't be abandoned if you don't have to deal with dynamic data.

[@realbss1]

@@RalfVogler : Elasticsearch is NOT a dumb search engine, it's a NoSQL database ! Beside that you can use several mathematical functions on fields, similar to a relational database. As MQTT as well is a typeless protocol elasticsearch would be a perfect match to store anything that would possibly arrive over MQTT. Along with Kibana it's quite easy to visualize the data, even other solutions ( including Grafana ) are possible. The answer to your questions is here : www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html, it can be expanded to any other critera the system is aware of ( time, name, field type, ... ). Given the fact that we talk about sensor data to be stored, mainly through MQTT, it might be worth looking at something like Elasticsearch that could easily handle . However, it's anyway recommended to use the DBMS you prefer an feel safe with ! Additionally it of course depends on the needs and given infrastructure. If you try to store (any) Windows eventlog data, especially from a whole domain, into a SQL database you'll understand the advantages of Elasticsearch and it's scaleability ... I'm just bringing it to the table, it's everyone's decission to have a closer look or not. My recommendation is : If you run just a couple of sensors use what you already have. If you want to scale up to hundreds of sensors you might have a closer look to Elasticsearch.

[@RalfVogler]

"use a dedicated field to indicate the type of the value" My point was that if you do that, you can't do things like average (which you are probably interested in for sensor values). Correct me if I'm wrong, I never used Elasticsearch. Of course you can do anything. Just writing to a file is also fast. I would assume that even for storing data of hundreds of sensors the choice of database doesn't matter. Where you will see a difference in performance is when you search in non-indexed fields, aggregate data over wrong types or use joins when the database doesn't support it. The built-in capabilities is the other important point: does Elasticsearch support querying time series data? Concerning types: either you declare the types for each MQTT topic or you let something like InfluxDB infer them from the first message.

[@michaelgruner941]

Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)

[@AndreasSpiess]

It was long overdue for me as I wanted to use it for my installation. I hope you had a good trip home!

[@trejohnson7677]

I’m glad I was already on the path of least resistance w/ Grafana

[@AndreasSpiess]

Maybe you have a look at the IOTstack project. There you find everything easier...

[@christosmoutevelis4896]

Thank you , always something to learn every week in an easy way

[@AndreasSpiess]

You are welcome!

[@ukaszs1349]

did you ever consider releasing t-shirt with "we like programming, but we love the result" - I'd buy it :D

[@AndreasSpiess]

Cool sentence! So fat I do not have any t-shirts or mugs. Maybe in the future...

[@SmithyScotland]

Thank you. I've been struggling for years with mysql and an inflexible php graphing library.

[@AndreasSpiess]

This is maybe a little more flexible...

[@PileofKyle]

You're awesome, thanks for sharing this knowledge. I've been learning a lot from you.

[@AndreasSpiess]

You are welcome!

[@luisd101]

Very nice! The problem with the Pi for DBs is the SD. Even good SDs tend to worn out fast and you lost your valuable DB. I would recomend to store db in other disk even at the expense of use USB disk. There are other SBC with SATA/PCIe that will be more suited this purpose.

[@AndreasSpiess]

I once did a video about that. For me so far I had no bad experience after more than a year (not Grafana,SQLite)

[@rbid]

Excellent video. Thanks. Greetings from the galilee

[@AndreasSpiess]

Glad you liked it!

[@Mariete]

I would like to see also a video about continuous queries and retention policies in detail with real examples related to weather (and possibly energy data). I have about three years of data every few seconds for my many weather and energy sensors but find very cumbersome to implement retention policies and continuous queries and to downsample the already existing data.

[@AndreasSpiess]

I do not see that this is a topic with lots of interested people. I think you will find some information on the net. Your situation is perfect as you can play with your data and find out which policy fits your needs

[@eduardogodoy7621]

Andreas did you stop numbering your videos ( #Number). The last one was #245. In my opinion, it facilitates finding specific videos and contents. I hope you continue using it. Thanks

[@AndreasSpiess]

No, but I always have about the newest 10 videos without numbers.

[@viniciusnoyoutube]

Fantastic job! Just what I was needing for my project. Thank you!

[@AndreasSpiess]

You are welcome!

[@AndersJackson]

Great video, as usual. But you should start to use the ip(8) command and not ifconfig(8), as that is obsolete and are going to be removed. And it is much more flexible and useful, as you don't need the arp(8), route(8) etc as that is all merged into the ip(8) command. And it is shorter to write. :-) ip -4 address show # can be shorted down to 'ip -4 a s' ip -6 address show ip route show # can be shorten down to 'ip r l' ip neighbour show # can be 'ip n s', also try 'ip -6 n s'

[@AndreasSpiess]

I did not know this command. I will try it. Thanks.

[@abrax5]

Really inspiring ideas, thank you! I'm thinking to move my solar panel yield visualization to an influxdb and grafana stack.

[@AndreasSpiess]

Good idea!

[@DavidL3430]

Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb. I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years! While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!

[@AndreasSpiess]

A other viewer commented he is using python scripts instead of Node-red

[@RalfVogler]

github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer

[@eriklaken1025]

Nice for my next step, 3 monts from now, i donate when i download the sd card, Have a Nice Day and Thank,S for so much Help!

[@AndreasSpiess]

You are welcome!

[@eriklaken1025]

Thank You, now i have made a sensor for my garden, Very litle solarpanel, a Mini_Pro that sleeps all Day with a Ds3231 clock, wakeup Once a Day send by WiFi with a ESP8266 with turn On with mosfet and give data to mqtt. The System works and have with 18650 batery power enoff. The py and MQTT make a Nice http:). So you can do a lot with a ESP8862 in this way. No power problem. I see On my computer How dry the garden is and maby give iT Some action.

[@asiw]

Happy birthday Andreas, hope you have a good one.

[@AndreasSpiess]

Yes, I had a good one. My new bike for the Danube tour arrived right that day :-)

[@partscience]

Wao!!!! Great idea!!!

[@AndreasSpiess]

:-)

[@rainbowforest09]

This is what i'm looking for. Thanks!!

[@AndreasSpiess]

:-)

[@desy666]

Useful and interesting as usual 😉

[@AndreasSpiess]

:-)

[@drfritz142]

Thank you. This is what I needed. I'll get to work right away...

[@AndreasSpiess]

Enjoy!

[@qamarkilani551]

Many Thanks Andreas - Well Done and hope to see more

[@AndreasSpiess]

You are welcome!

[@Aemilindore]

Thanks so much for providing the SD image. Thanks thanks thanks

[@AndreasSpiess]

You are welcome!

[@christopheralexander8763]

Great video, Andreas. Really enjoyed this one.

[@AndreasSpiess]

Thank you!

[@kwazar6725]

Cool. Great stuff Andreas.

[@kwazar6725]

Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.

[@AndreasSpiess]

Thank you!

[@joakimbergstedt4390]

Hi! Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)

[@stefanmarinescu5086]

That is too complex for my level but is very nice to know that if needed, this is a good reference for learning.

[@AndreasSpiess]

It should be easy if you install the image of the SD card to start playing and learning...

[@avejst]

Fantastic project👍😀 Thanks for sharing😀👍

[@AndreasSpiess]

You are welcome!

[@midafricam]

Another big thank you, I have been working thru this tutorial on my VirtualBox (Installed Ubuntu 18.04.3 LTS). No RPi yet or Arduino, ordered the LILYGO® TTGO T-Beam ESP32 , will order the other kit later on. Create a LoraNode to test with and log on TTNMapper. Your persistence as a KZbin creator assisted me in SouthAfrica to learn alot in a short space of time. Thank Andreas Just need to go thru Grafana graph to see why my Voltages capture from a Mikrotik RB3011 are not being extracted from the InfluxDB. Use the Node-Red Mikrotik (node-red-contrib-mikrotik3), this connects via the Mikrotik API and extracts the voltages and temp (from RB3011) Inject /system/health/print into the Mikrotik node - then into InfluxDB. Operating in the Wireless/Fiber ISP industry we have so many different monitoring systems UNMS, Dude, Nagios, Observuim this solution is a perfect method Node-Red -> InfluxDB -> Grafana (now need to investigate capturing SNMP traps and displaying them in Grafana) At least we can know monitor our repeater sites more proactively with movement sensors (from a universal dashboard) , batteries & solar panels are hot items on the "thieves shopping list"

[@AndreasSpiess]

Maybe you watch also the video about docker. There, another South African helped me to get all the gadgets you use onto docker containers... I think this kind of tools is ideal for your needs. Fast and cheap...

[@jensriebold2818]

Hi Andreas, vielen vielen Dank für das Video - Ich setze das gerade auf..... SUPER!!! vielen Dank!!! vy 73 de Jens, DG1OAM

[@AndreasSpiess]

Vielleicht schaust du dir noch die Videos über IOTstack an. Da wird der ganze Setup schnell und einfach...

[@edsteve]

In the measurement "stations". Wouldn't it be the better option (faster for queries) to store the "name" as tag and not as field? Tags are indexed and fields aren't. Still starting with InfluxDB but that's how i understand it.

[@AndreasSpiess]

I am not a specialist in InfluxDB and my data size is small. So I never investigated.

[@byteborg]

@Andreas did you consider writing the name field not as a measurement but as a tag? This is achieved by writing not one object as msg.payload to the InfluxDB node, but writing a list [] of two objects, first one with the measurements, second one with the tags. If you set the name field as a tag, it should also show up automatically as a completion choice in Grafana.

[@AndreasSpiess]

I did this after the video. It is easier for the selection in the dashboard. But during the take I did not know how to do it :-(

[@g.s.3389]

you'd better warn that grafana works on RPI3 only and not on previous raspberry boards.

[@AndreasSpiess]

I did not know that.

[@nicklaspersson4687]

Thanks for warning Giuseppe Suanno. I was about to do this on a zero because i happen to have one available. Even if i suspected it might need a gruntier board like PI3. Why only on Pi3? Can you please elaborate or provide source?

[@ajaybnl]

I am using it on orangepi lite.

[@kiowablue2862]

I've got it running on a RasPi 2, and have also ran it on the original 512MB RasPi Model B. Granted it's slower on the 1st generation Pi, but it runs with no unreasonable difference in speed on a Pi 2 compared to a Pi 3. (which I've also run it on)

[@encephalophiliac]

Hi Andreas! Love this video, and thanks for putting together the image. I recently ordered a few Rasperry Pi Zero 2 Ws, and I'm trying to load the image you provided to one of them. When I load the flashed SD card and plug in the power, the device doesn't boot. No power light, no indication that it's connected to wifi (after adding wpa_supplicant.conf and SSH to the root directory). Is there any reason the image as provided wouldn't work on a raspi zero 2 W? Thanks again!

[@AndreasSpiess]

I do not know if it works on the 2W. Two things to consider: 1. The memory of the 2W is very low for such applications. I would go for at least 2G 2. Maybe you watch the video about IOTstack. There you find another way of installing all the goodies...

[@icecaper]

Great video again, Andreas and superb timing as I too have a project which requires this type of sensor recording/display. Have you used an IOT2020?

[@AndreasSpiess]

No, I usually do not use such "professional" components like the IOT2020

[@DaveBrown1]

Excellent, very good.

[@AndreasSpiess]

Thank you!

[@daffdaffelito]

A verry good video! Thanks. I all ready got influxdb and node red on my server. But i never used the influxdb node in node red befor. It would be great if your node red flows where published somewhere!!

[@AndreasSpiess]

They are: In the comments, you should find a link.

[@daffdaffelito]

@@AndreasSpiess thanks! Looked a bit better now and found them.

[@stephsoltesz6731]

Good Day again ! Gosh I love your Video's and so timely at that ! After watching several of your vid's I'm struck by this particular one (#255) which answers many of my needs as to a project I want to get started on BUT some insight from you would be most helpful, I'm sure. The intended project, is to access my Solar System Components which include a Midnite Classic 200 MPPT Controller, a Samlex EVO-4024 Inverter/Charger using ModBus, pull stats & config info and provide a web based interface to view the info in Real Time and with full long term logging. In addition to that, I would like to setup a Weather Station (wind, rain etc) and a "Solar Monitor" of some sort that tells me the strength of the sun at points in the day (located with my fixed, ground mounted solar panels). Although I've never seen a "Solar Strength Sensor" as such, so this might be "interesting". Ultimately, I would like to be able to incorporate various Solar devices (brands) into a database which would allow people to be able to access the same info from their various devices. Something which can be distributed as a complete application or an image/iso/? I intend this project to be opensource / freeware in the Best Interests of everyone. Yup, I'm an RE Greeny ! LOL Any suggestions, thoughts or recommendations you have would be greatly appreciated ! Any recommended "Weather Stations" which can be monitored by a Pi would be helpful, as I'm sure some are better (easier) than others. Thanks in Advance for your thoughts & consideration Steve

[@AndreasSpiess]

Maybe you slice this big elephant into smaller parts and start small. Then you can add all your bells and whistles as you go on. You will most probably make some mistakes and learn a lot on the way.

[@stephsoltesz6731]

@@@AndreasSpiess Very wise advice but being a IT Systems guy for 30 some years I have no fear of bigger projects with a longer view. ;-) Things have worked out wonderfully actually and again, thanks to your video's, you made my life so much easier so THANKS AGAIN ! I am now effectively communication with the Midnite Classic-200 Controller (Modbus via TCPIP) and writing data to Influx and then to Grafana. What a Great set of tools ! I have just received the confidential materials from Samlex (had to sign NDA's ;-( ) and purchased a USR-IOT RS485-> TCPIP converter (I maybe should have gotten an RS485 hat but I wanted some extra functionality is it handles RS232 as well_) so soon, I will be able to interface with the Inverter/Charger. Next stage to go with this is to setup a 12VDC -> 5VDC bus, from my Battery Bank to power the RasPi, USR Converter & a 9VDC line for the Router which would let them run outside of the solar system direct from the battery bank. Weather Station will follow in due course when I have time.

[@tremartian3803]

I suppose all the credentials should be changed to use the SD card image securely? What kind of changes should be done? Paswords for the Raspberry Pi, Node_red, influxDB and Grafana should be changed? It would be nice to have a guide how to do this and how changing the password of one app affects to another app.. in other words, how to keep the system working after passwords are changed.

[@AndreasSpiess]

You can decide what you want to change. Because the server is behind my firewall I leave them as they are.

[@leedevries5660]

Great work again Andreas!

[@AndreasSpiess]

Thank you!

[@aliasname5518]

Hello Andreas, does this work with an DHT22 Sensor or an Adafruit Sensirion SHT31-D? Where would I need to set the sensors up, in Node-Red? Thank you - great video!

[@AndreasSpiess]

I usually do not connect sensors to the Raspberry because they have to be at different places. I use ESP32s and connect them with node-red via MQTT.

[@Techonsapevole]

Thanks Andreas, did you consider to wrap this config in Docker/Docker Compose instead SD ? It would be easier to upgrade

[@AndreasSpiess]

No, not for the moment. Maybe later...

[@PhG1961]

Waw, excellent work ! Heavy stuff aswell !

[@AndreasSpiess]

The simple stuff is done by others ;-)

[@LucasHartmann]

Playing with influx and grafana for computer (cpu ram ssd-cache and disk usage) and backup status monitoring. Got it working with docker containers after a few hours tinkering. Did not see the need for node red, though. Using curl for posting, and trying to get raspberry+Screenly working for display.

[@AndreasSpiess]

I like node-red because I know it ;-) there are many other possibilities to bridge from MQTT into influxDB. Did you try docker to use GPIOs or Serial on the RPi?

[@LucasHartmann]

@@AndreasSpiess I used docker for influx and grafana, not for the data collector /sensor program. You may be able to use gpio and serial inside a container by usingal a privileged container and mounting /sys and /dev/tty*** as volumes on it.

[@nikolaimorozov6383]

Many thanks for your video!

[@AndreasSpiess]

You are welcome!

[@tarmiziizzuddin337]

Thank you very much sir! Always love your video

[@AndreasSpiess]

You are welcome!

[@user-lj9uu9zb4w]

Thank you for the hundreds of really useful, interesting, up to date tutorials and info, you're pure gold! I have a question regarding the pi node-red server side, i need to have it in a distant location and feed it with 12V batteries (car battery) and let it run node-red influx db and grafana. What would be a good way to convert 12V down to 5V for the pi? Would one of these cheap buck converters do? What do you suggest? thank you in advance!

[@AndreasSpiess]

A buck converter with at least 3A should do it. There are many projects around if you search with Google