"We like programming, but we love result" Quote of the day. Thank you sir
@AndreasSpiess5 жыл бұрын
You are welcome!
@bigheat865 жыл бұрын
Even I can confirm, and I'm a developer.
@lightpixeldotnet4 жыл бұрын
this :]
@richard_wenner5 жыл бұрын
Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).
@AlenTuzlak5 жыл бұрын
I agree 100%. Andreas is amazing!! Thank you so much Sir for sharing knowledge with everyone and great commitment on making amazing videos !!
@AndreasSpiess5 жыл бұрын
We are slow here. That is probably why our time runs a little slower... I hope the content still helps you a little for your pollution project!
@richard_wenner5 жыл бұрын
@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.
@AndreasSpiess5 жыл бұрын
Security is not my specialty, fortunately ;-)
@aux7895 жыл бұрын
Amazing! I had to stop the video on two occasions I was so overwhelmed by the sheer power of this suite of FREE software! Thanks Andreas!
@AndreasSpiess5 жыл бұрын
That is the power of KZbin. You can use it at your pace ;-)
@voltlog5 жыл бұрын
This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.
@AndreasSpiess5 жыл бұрын
I think it is not so easy if you have to do everything from the ground. So we both were happy we were able to stand on the shoulders of others ;-)
@chuxxsss5 жыл бұрын
I did find another script from another person with red-node. It is handy for solar and batteries. I hated manually install parts of Peters script the .npm title. But Peter has done a great Job with this script with Mr Sharky. I love it. Peter is a great person thank goodness we have him around. The graphics I think was made by him too.
@AndreasSpiess5 жыл бұрын
With his new version, you should not need any manual install...
@hsmptg5 жыл бұрын
Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!
@AndreasSpiess5 жыл бұрын
I mentioned the compression, but not in detail. I do not plan a video on that because it is very special. Maybe once part of another one. Who knows?
@PeterScargill5 жыл бұрын
Nice to see you're making good use of "the script" Andreass.
@AndreasSpiess5 жыл бұрын
I always like to stand on the shoulders of big men ;-) Makes me more productive! Like that, everybody can profit.
@PeterScargill5 жыл бұрын
@@AndreasSpiess Hah. I do that all the time. My script enables software like Node-Red, written by people cleverer than me!
@deangreenhough34795 жыл бұрын
Thank you both for your wonderful work. I have a beautiful image on my Pi and it’s all working 😁🏴
@AndreasSpiess5 жыл бұрын
The timestamps of the example data are from 2019-02-06. Please choose an appropriate date range in Grafana if you want to display them
@springwoodcottage42485 жыл бұрын
Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!
@AndreasSpiess5 жыл бұрын
You are welcome. And I agree that retention strategies are not easy to define these days with very cheap disk space...
@opsahle25 жыл бұрын
Extremely usefull video - I am very happy to be one of your Patrion-supporters! Keep it coming, Andreas!
@AndreasSpiess5 жыл бұрын
Thank you for your support! It is always good to read that my videos are useful...
@rgrf3 жыл бұрын
Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.
@AndreasSpiess3 жыл бұрын
Maybe you also watch my videos about IOTstack...
@rgrf3 жыл бұрын
@@AndreasSpiess I did Andreas and for the first time doing useful things with this stack. More thanks. My stuff is at rogerfrost.com. Best wishes.
@hoekbrwr5 жыл бұрын
This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!
@AndreasSpiess5 жыл бұрын
I transferred historical data exporting to CSV and creating the needed input strings for influxDB with Excel.
@yura9795 жыл бұрын
I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O
@AndreasSpiess5 жыл бұрын
Maybe you can save some time now ;-)
@Argosh5 жыл бұрын
I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!
@AndreasSpiess5 жыл бұрын
I hope you will be successful in transferring the data from your source into InfluxDB.
@lucapesce33355 жыл бұрын
Cool, I use InfluxDB for my home, but never understood the retention policy, thank you. I have three point about your setup: - InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination - This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this) - All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover
@AndreasSpiess5 жыл бұрын
1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot. 2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.
@noealves84005 жыл бұрын
Mr Spiess, my deepest gratitude for your work and for sharing great content in a video like this. I became your Patreon today to support your work!
@AndreasSpiess5 жыл бұрын
Thank you for supporting the channel. I appreciate it!
@mrtarantl5 жыл бұрын
Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...) Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) ) Greetings from Austria
@AndreasSpiess5 жыл бұрын
Thank you! Jokes are not easy in an international environment. Glad to read that you like them. I like music, but in videos like that, I hate it myself. This is why I do not use it...
@ursmii_4 жыл бұрын
I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video
@AndreasSpiess4 жыл бұрын
Very fast! Did you try the docker Installation?
@ursmii_4 жыл бұрын
no I did not. I'll try to create a surveillance for 24/7 running PCs (if message does not arrive periodically - alert)@@AndreasSpiess
@AndreasSpiess4 жыл бұрын
Interesting project! We created a whole project with lots of useful containers (IOTstack)...
@santorcuato5 жыл бұрын
I have no words Andreas. Astonishing. Thanks.
@AndreasSpiess5 жыл бұрын
You are welcome!
@Alby_VA3 жыл бұрын
Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,
@AndreasSpiess3 жыл бұрын
I also like Grafana...
@Alby_VA3 жыл бұрын
@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)
@mikethezipper5 жыл бұрын
You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.
@AndreasSpiess5 жыл бұрын
Glad to read that!
@LucasGuillemette5 жыл бұрын
I'm a programming student and I was this kind of data retention policy saves a lot of work. Glad I don't have to reinvent the wheel.
@AndreasSpiess5 жыл бұрын
We live in a good time with all this open source software...
@UpLateGeek5 жыл бұрын
I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.
@AndreasSpiess5 жыл бұрын
MongoDB seems to use quite a lot of disk space because it is made for other purposes. There is a comparison on the influxDB page...
@UpLateGeek5 жыл бұрын
@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.
@sausagehider5 жыл бұрын
Such a coincidence. I was up until 1:30 am this morning looking at almost the exact same thing. Excellent content, I will give it a go today. Thanks. Martin.
@AndreasSpiess5 жыл бұрын
Good luck. I hope it will work for you!
@MirkWoot5 жыл бұрын
Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.
@AndreasSpiess5 жыл бұрын
I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)
@steveb76005 жыл бұрын
You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.
@portfedh3 жыл бұрын
Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.
@AndreasSpiess3 жыл бұрын
Excellent!
@brunosolothurnmann92054 жыл бұрын
Thank you - I installed Grafana and InfluxDB on my existing Raspberry Pi 3+ with an already installed Node-Red and Mosquitto. For the installation of Grafana and InfluxDB I found some instructions on the internet. I use Node-Red and Mosquitto a lot for my home automation and used until now SQLite or flat files. Your Video helped me to setup it - and it runs fine. Now I'm going to adapt my home automation to the new data sources.
@AndreasSpiess4 жыл бұрын
Now you can watch the newer version (dockerized) of this setup.
@YouGenom5 жыл бұрын
Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)
@AndreasSpiess5 жыл бұрын
So far I always used Node-Red and do not know Home Assistant.
@YouGenom5 жыл бұрын
@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.
@kokoscom5 жыл бұрын
Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)
@kokoscom5 жыл бұрын
it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)
@AndreasSpiess5 жыл бұрын
25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...
@followthetrawler4 жыл бұрын
Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun. Thanks for another great video and the idea...
@AndreasSpiess4 жыл бұрын
There are a few other videos about weather station hacking and Pi and Docker (IOTstack)
@thorsten92114 жыл бұрын
Thank You! I love how well structured your videos are. Suits my taste as german engineer ;)
@AndreasSpiess4 жыл бұрын
Thank you! Maybe you also watch the one about Docker. It simplifies the installation of those programs
@thorsten92114 жыл бұрын
@@AndreasSpiess Yes i did and everything is working nicely together. :)
@fpaolo634 жыл бұрын
Hi Andreas, just to share. I created a "home heating system monitor" based on some arduino-nano, reading NTCs/DS1820, and raspberry interconnected by field-bus based on modbus (RS485). Raspberry is running a python script to collect data and store it on influxbd, Grafana is used to visualize data and correlate info. GOAL: integrate solar PV, heat pump system and Gas heat generator, control floor heating and try to be more green. ... will see...
@AndreasSpiess4 жыл бұрын
It sounds like an interesting project. Maybe you have also planned to add something like hass.io? And maybe you also watch the video about IOTstack (if not already done)
@fpaolo634 жыл бұрын
... Yes I am looking deep in Hass.io ... but ... before go in Strong IOT every one should read 'Abusing the Internet of Things'. For now l prefer to stay with a well lock down reporting tools.
@laveur5 жыл бұрын
Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.
@AndreasSpiess5 жыл бұрын
At least you would have great support for your project. Not as we "simple people" ;-)
@px16905 жыл бұрын
Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.
@AndreasSpiess5 жыл бұрын
Maybe I have to look at hass.io once...
@ajaybnl5 жыл бұрын
Everything new. I had watched twice to understand. I want this setup too for my home sensors. Thank you dear.
@AndreasSpiess5 жыл бұрын
You are welcome!
@ajaybnl5 жыл бұрын
Im now successfully using grafana influxdb with mosquitto (EspMqtt) using node red on my OrangePi Lite.
@klogaraa5 жыл бұрын
Amzaing effort from you and Pete's , keep the videos coming Andreas 👍
@AndreasSpiess5 жыл бұрын
Thank you!
@MrAtarifreak5 жыл бұрын
Thank you for your great video. These are the projects that I can learn from a lot....
@AndreasSpiess5 жыл бұрын
You are welcome!
@regenerativelife9915 жыл бұрын
thanks a lot for this research and experimentation. we will definitely use this for our farm.
@AndreasSpiess5 жыл бұрын
I hope your application will add some value :-)
@asiw5 жыл бұрын
This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on KZbin. Your viewers might find the comparison interesting.
@AndreasSpiess5 жыл бұрын
SQL is always a possibility. It is stable and does its job.
@squalazzo5 жыл бұрын
@Andreas issues are now solved, as you know by latest comments on Pete's blog... i'm MrShark :)
@AndreasSpiess5 жыл бұрын
Thanks! The video was already out then. I pin your comment that everyone can read it.
@squalazzo5 жыл бұрын
@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine
@squalazzo5 жыл бұрын
@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)
@MrJanulis5 жыл бұрын
@@squalazzo Influxdb node still requires manual installation - tried just a couple of hours ago
@realsushi_official11163 жыл бұрын
Very interesting and easy to jump in. Actually I was using a Python Jupiter Notebook for Receiving, parsing, sending payload to the MQTT broker. This solution is great, especially for future maintenance
@AndreasSpiess3 жыл бұрын
I never thought about using Jupiter Notebook for something like that. Now I have to look into it. Tanks for the info.
@familyplans37884 жыл бұрын
just putting up xmas lights but as soon as that is done i will be following this video and cant wait to use Mr. Scargills 'script'
@AndreasSpiess4 жыл бұрын
Maybe you watch the „ultimate pi server“ video first.
@familyplans37884 жыл бұрын
@@AndreasSpiess LOL i was just wondering that !! good job the mrs LOVES your accent so i get away with watching geeky videos
@alanjrobertson2 жыл бұрын
Really helpful explanation of the benefits of time series databases for IoT, thank you 👍👍
@AndreasSpiess2 жыл бұрын
Glad it was helpful!
@MitchellWongHo5 жыл бұрын
Excellent demonstration of how little you need to collect and visualise data. I have a spare PINE64 which I'd like to try using these building blocks
@AndreasSpiess5 жыл бұрын
You have to check which operating systems the script supports if you want to use it.
@veljac995 жыл бұрын
Nice video Andreas! In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before). Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting. I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon. Point is that any additional knowledge nicely fits in picture. Constant evolution...
@AndreasSpiess5 жыл бұрын
Other commenters also mentioned that it is easy with Home assistant. Maybe I have to try it one...
@ThWischi2 жыл бұрын
Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas
@AndreasSpiess2 жыл бұрын
Thank you for your kind words!
@einars8995 жыл бұрын
Interesting and useful video! It seems we are heading back to remote sensing? I like that.
@AndreasSpiess5 жыл бұрын
Sensing and acting based on these data is IOT for me ;-)
@wilman_studio5 жыл бұрын
Awesome guide to getting Grafana up and running, thank you :)
@AndreasSpiess5 жыл бұрын
You are welcome!
@princebanini4 жыл бұрын
this was an eye opener for me. always a good time learning from you
@AndreasSpiess4 жыл бұрын
Glad to hear it!
@wassfila5 жыл бұрын
I love it, now we really talk IoT. One more step Andreas, automated MQTT to influx converter. Let me know if you'd like to have the links. In my porject (google "home smart mesh"), I have such a python converter. It subscribes to a certain MQTT topic pattern, and posts jsons as is into influxdb. Everyone is already very familiar with MQTT directly produced from ESP and others, with an automated converter, you would not need to edit a custom translator for every sensor you add. Yeah and I forgot something, once MQTT is plugged to influxdb and grafana, you enpower it with something like zigbee2mqtt and have awesome cheap market products on you dashboard !
@AndreasSpiess5 жыл бұрын
I would be interested in the links. Did you do a tutorial? So far I always thought that ZigBee is quite expensive for makers and not well accepted in our community. Maybe I am wrong...
@DavidL34305 жыл бұрын
Hello, I spent a quick 30 min digging into your “home smart mesh” with the objective of understanding just the mqtt to influx converter. I feel close to getting it but honestly got a little lost in trying to peel the mesh application out of it. In my case I have at least a dozen esp and other devices all speaking to mqtt over Wi-Fi to node red then over to RRD databases. I want to explore updating this to mqtt to influx to grafana directly. Examples: /sonoff/1/temperature/ or /base/55/humidity/ or /1wire/“serialnumber”/“englishID”/temperature/ Do I configure all these in JSON or just create rules for each generic example? And Bravo to Peter and team for that script which I have used several times on Pi and other Linux installs!!
@RalfVogler5 жыл бұрын
Haven't tried it yet, but there's a Telegraf plugin for MQTT to Influx: github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer
@wassfila5 жыл бұрын
@@AndreasSpiess for MQTT to influx with python please find a direct link to the github influx python directory here github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx I do not have a special tutorial for the MQTT to influx, but to be honest I struggle to make a documentation or tutorials as clear as you do, you are better at that Andreas :) Nevertheless I spent effort to document the whole "Home Smart Mesh" which has a lot of custom RF in it, and it is not easy to isolate the raspberry pi scripts from the rest. I tried to document on Hackaday hackaday.io/project/20388-home-smart-mesh and on Github which is more focused on the nRF52 Hardware github.com/nRFMesh/nRF52_Mesh When it comes to Zigbee, I can confidently say that it became cheap enough now with all sort of ready sensors ~< 10 € from Ali, and an official nRF52840-Dongle < 10€ from Mouser. Now I would agree with you if you say nRF52 Thread or Zigbee is not well supported by Arduino, but Andreas, we have to make it happen, by selecting best HW and pushing it to be come easy and popular. I started wrapping the NRF SDK official from Nordic around a c++ API with classes that make it Arduino like, my latest progress on that here github.com/Roblibs/rover_firmware/blob/master/applications/03_sensored_bldc/main.cpp but far from being a real arduino core. The nRF52840 is I believe a real challenger for the ESP32, as the nRF52 has native USB, an Automated Programmable Peripheral Interconnect, and with its custom radio I achieve latency ~ 1 ms. I'll be glad to share any required further info and details.
@wassfila5 жыл бұрын
@@DavidL3430 please find a direct link to the python script that converts MQTT to influx, it should be possible to take that directory away from all the rest github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx But I admit, it is not packaged to be used without Python knowledge, as you notice, even if I have the main config in the config.json, a lot of the logic is coded in the def mqtt_on_message, for example the number of words in the topic e.g. mesh/node5/temperature and also I force some types to be float or int in lines 36 to 43. But on the good side, I even prepared scripts for backup and export, and the note that the config can be different depending on the hstname so that you have many raspberry pi and save all config in the same directory with names config_rasphostname1.json, config_rasphostname2.json, ... I will gladly provide you any further help if you want.
@jab55693 жыл бұрын
Nice work Andreas! Thank you for putting this together, very informative.
@AndreasSpiess3 жыл бұрын
My pleasure!
@manuelorellanahinojosa73564 жыл бұрын
Your channel is pure gold!! Greetings from Chile!
@AndreasSpiess4 жыл бұрын
Thank you very much!
@athanasiosstefanopoulos52115 жыл бұрын
What another great video, thank you Andreas! Your video inspired me for changing my implementation on how to use historical data.
@AndreasSpiess5 жыл бұрын
I hope you will be successful!
@athanasiosstefanopoulos52115 жыл бұрын
@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana. And all these thanks to you again :)
@realbss15 жыл бұрын
Hi Andreas, relational databases do not necessarily have to be setup with fields like "Temp, Pressure, ...", you can as well do an different approach and use a dedicated field to indicate the type of the value. Something like "Timestamp, Type, Value", that way the only restriction is the type of value you want to store ( integer, float, string, ... ). If you want to get completely rid of that you should have a look at elasticsearch, it's a database system that is highly scalable and very dynamic. You can easily scale it up from one DB node to hundreds in seconds, simply by adding additional nodes and the same way back (in reality there is a little downside in downsizing - let me know if you are interested). Elasticsearch does NOT stick on types, or even database table schemas. It simply stores what you send - if there is no - for example Temperature - it just creates it dynamically for you. Additionally the values are the same flexible - no need to preselect a type like float or a length, it's just dynamically created on the fly. It's even able to store >100k Messages/s depending on the machine it's running on. In reality I had soe time ago a DB running where I was feeding event log data from 9 Active Directory domain controllers and keeping ~1 year of history of that data. Peaks where around 500k messages/s and around 5-7 million entries a day on a 6 year old reused hardware. Basically a perfect match for MQTT where we do not have strict type/name rules as well.
@AndreasSpiess5 жыл бұрын
You are right. You can store these primitive sensor values in every database without any problems. AFAIK Elasticsearch is considered to be a "NoSQL" database. They are very good at certain things like you describe: Unstructured data and also full-text search. As I said, the strength of time-series databases is not the storage side but on the retention as well as the extraction of time-based data. I do not want to focus on database concepts on this channel because there are people which are much more capable than me in this matter.
@RalfVogler5 жыл бұрын
It's a search engine, why should you use it as a database? How do you efficiently get the average of a field (or even better over a time period)? Types and structure shouldn't be abandoned if you don't have to deal with dynamic data.
@realbss15 жыл бұрын
@@RalfVogler : Elasticsearch is NOT a dumb search engine, it's a NoSQL database ! Beside that you can use several mathematical functions on fields, similar to a relational database. As MQTT as well is a typeless protocol elasticsearch would be a perfect match to store anything that would possibly arrive over MQTT. Along with Kibana it's quite easy to visualize the data, even other solutions ( including Grafana ) are possible. The answer to your questions is here : www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html, it can be expanded to any other critera the system is aware of ( time, name, field type, ... ). Given the fact that we talk about sensor data to be stored, mainly through MQTT, it might be worth looking at something like Elasticsearch that could easily handle . However, it's anyway recommended to use the DBMS you prefer an feel safe with ! Additionally it of course depends on the needs and given infrastructure. If you try to store (any) Windows eventlog data, especially from a whole domain, into a SQL database you'll understand the advantages of Elasticsearch and it's scaleability ... I'm just bringing it to the table, it's everyone's decission to have a closer look or not. My recommendation is : If you run just a couple of sensors use what you already have. If you want to scale up to hundreds of sensors you might have a closer look to Elasticsearch.
@RalfVogler5 жыл бұрын
"use a dedicated field to indicate the type of the value" My point was that if you do that, you can't do things like average (which you are probably interested in for sensor values). Correct me if I'm wrong, I never used Elasticsearch. Of course you can do anything. Just writing to a file is also fast. I would assume that even for storing data of hundreds of sensors the choice of database doesn't matter. Where you will see a difference in performance is when you search in non-indexed fields, aggregate data over wrong types or use joins when the database doesn't support it. The built-in capabilities is the other important point: does Elasticsearch support querying time series data? Concerning types: either you declare the types for each MQTT topic or you let something like InfluxDB infer them from the first message.
@michaelgruner9415 жыл бұрын
Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)
@AndreasSpiess5 жыл бұрын
It was long overdue for me as I wanted to use it for my installation. I hope you had a good trip home!
@trejohnson76773 жыл бұрын
I’m glad I was already on the path of least resistance w/ Grafana
@AndreasSpiess3 жыл бұрын
Maybe you have a look at the IOTstack project. There you find everything easier...
@christosmoutevelis48965 жыл бұрын
Thank you , always something to learn every week in an easy way
@AndreasSpiess5 жыл бұрын
You are welcome!
@ukaszs13493 жыл бұрын
did you ever consider releasing t-shirt with "we like programming, but we love the result" - I'd buy it :D
@AndreasSpiess3 жыл бұрын
Cool sentence! So fat I do not have any t-shirts or mugs. Maybe in the future...
@SmithyScotland5 жыл бұрын
Thank you. I've been struggling for years with mysql and an inflexible php graphing library.
@AndreasSpiess5 жыл бұрын
This is maybe a little more flexible...
@PileofKyle4 жыл бұрын
You're awesome, thanks for sharing this knowledge. I've been learning a lot from you.
@AndreasSpiess4 жыл бұрын
You are welcome!
@luisd1015 жыл бұрын
Very nice! The problem with the Pi for DBs is the SD. Even good SDs tend to worn out fast and you lost your valuable DB. I would recomend to store db in other disk even at the expense of use USB disk. There are other SBC with SATA/PCIe that will be more suited this purpose.
@AndreasSpiess5 жыл бұрын
I once did a video about that. For me so far I had no bad experience after more than a year (not Grafana,SQLite)
@rbid3 жыл бұрын
Excellent video. Thanks. Greetings from the galilee
@AndreasSpiess3 жыл бұрын
Glad you liked it!
@Mariete5 жыл бұрын
I would like to see also a video about continuous queries and retention policies in detail with real examples related to weather (and possibly energy data). I have about three years of data every few seconds for my many weather and energy sensors but find very cumbersome to implement retention policies and continuous queries and to downsample the already existing data.
@AndreasSpiess5 жыл бұрын
I do not see that this is a topic with lots of interested people. I think you will find some information on the net. Your situation is perfect as you can play with your data and find out which policy fits your needs
@eduardogodoy76215 жыл бұрын
Andreas did you stop numbering your videos ( #Number). The last one was #245. In my opinion, it facilitates finding specific videos and contents. I hope you continue using it. Thanks
@AndreasSpiess5 жыл бұрын
No, but I always have about the newest 10 videos without numbers.
@viniciusnoyoutube5 жыл бұрын
Fantastic job! Just what I was needing for my project. Thank you!
@AndreasSpiess5 жыл бұрын
You are welcome!
@AndersJackson4 жыл бұрын
Great video, as usual. But you should start to use the ip(8) command and not ifconfig(8), as that is obsolete and are going to be removed. And it is much more flexible and useful, as you don't need the arp(8), route(8) etc as that is all merged into the ip(8) command. And it is shorter to write. :-) ip -4 address show # can be shorted down to 'ip -4 a s' ip -6 address show ip route show # can be shorten down to 'ip r l' ip neighbour show # can be 'ip n s', also try 'ip -6 n s'
@AndreasSpiess4 жыл бұрын
I did not know this command. I will try it. Thanks.
@abrax55 жыл бұрын
Really inspiring ideas, thank you! I'm thinking to move my solar panel yield visualization to an influxdb and grafana stack.
@AndreasSpiess5 жыл бұрын
Good idea!
@DavidL34305 жыл бұрын
Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb. I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years! While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!
@AndreasSpiess5 жыл бұрын
A other viewer commented he is using python scripts instead of Node-red
Nice for my next step, 3 monts from now, i donate when i download the sd card, Have a Nice Day and Thank,S for so much Help!
@AndreasSpiess5 жыл бұрын
You are welcome!
@eriklaken10255 жыл бұрын
Thank You, now i have made a sensor for my garden, Very litle solarpanel, a Mini_Pro that sleeps all Day with a Ds3231 clock, wakeup Once a Day send by WiFi with a ESP8266 with turn On with mosfet and give data to mqtt. The System works and have with 18650 batery power enoff. The py and MQTT make a Nice http:). So you can do a lot with a ESP8862 in this way. No power problem. I see On my computer How dry the garden is and maby give iT Some action.
@asiw5 жыл бұрын
Happy birthday Andreas, hope you have a good one.
@AndreasSpiess5 жыл бұрын
Yes, I had a good one. My new bike for the Danube tour arrived right that day :-)
@partscience5 жыл бұрын
Wao!!!! Great idea!!!
@AndreasSpiess5 жыл бұрын
:-)
@rainbowforest095 жыл бұрын
This is what i'm looking for. Thanks!!
@AndreasSpiess5 жыл бұрын
:-)
@desy6665 жыл бұрын
Useful and interesting as usual 😉
@AndreasSpiess5 жыл бұрын
:-)
@drfritz1425 жыл бұрын
Thank you. This is what I needed. I'll get to work right away...
@AndreasSpiess5 жыл бұрын
Enjoy!
@qamarkilani5515 жыл бұрын
Many Thanks Andreas - Well Done and hope to see more
@AndreasSpiess5 жыл бұрын
You are welcome!
@Aemilindore5 жыл бұрын
Thanks so much for providing the SD image. Thanks thanks thanks
@AndreasSpiess5 жыл бұрын
You are welcome!
@christopheralexander87635 жыл бұрын
Great video, Andreas. Really enjoyed this one.
@AndreasSpiess5 жыл бұрын
Thank you!
@kwazar67255 жыл бұрын
Cool. Great stuff Andreas.
@kwazar67255 жыл бұрын
Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.
@AndreasSpiess5 жыл бұрын
Thank you!
@joakimbergstedt43904 жыл бұрын
Hi! Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)
@stefanmarinescu50865 жыл бұрын
That is too complex for my level but is very nice to know that if needed, this is a good reference for learning.
@AndreasSpiess5 жыл бұрын
It should be easy if you install the image of the SD card to start playing and learning...
@avejst5 жыл бұрын
Fantastic project👍😀 Thanks for sharing😀👍
@AndreasSpiess5 жыл бұрын
You are welcome!
@midafricam4 жыл бұрын
Another big thank you, I have been working thru this tutorial on my VirtualBox (Installed Ubuntu 18.04.3 LTS). No RPi yet or Arduino, ordered the LILYGO® TTGO T-Beam ESP32 , will order the other kit later on. Create a LoraNode to test with and log on TTNMapper. Your persistence as a KZbin creator assisted me in SouthAfrica to learn alot in a short space of time. Thank Andreas Just need to go thru Grafana graph to see why my Voltages capture from a Mikrotik RB3011 are not being extracted from the InfluxDB. Use the Node-Red Mikrotik (node-red-contrib-mikrotik3), this connects via the Mikrotik API and extracts the voltages and temp (from RB3011) Inject /system/health/print into the Mikrotik node - then into InfluxDB. Operating in the Wireless/Fiber ISP industry we have so many different monitoring systems UNMS, Dude, Nagios, Observuim this solution is a perfect method Node-Red -> InfluxDB -> Grafana (now need to investigate capturing SNMP traps and displaying them in Grafana) At least we can know monitor our repeater sites more proactively with movement sensors (from a universal dashboard) , batteries & solar panels are hot items on the "thieves shopping list"
@AndreasSpiess4 жыл бұрын
Maybe you watch also the video about docker. There, another South African helped me to get all the gadgets you use onto docker containers... I think this kind of tools is ideal for your needs. Fast and cheap...
@jensriebold28183 жыл бұрын
Hi Andreas, vielen vielen Dank für das Video - Ich setze das gerade auf..... SUPER!!! vielen Dank!!! vy 73 de Jens, DG1OAM
@AndreasSpiess3 жыл бұрын
Vielleicht schaust du dir noch die Videos über IOTstack an. Da wird der ganze Setup schnell und einfach...
@edsteve4 жыл бұрын
In the measurement "stations". Wouldn't it be the better option (faster for queries) to store the "name" as tag and not as field? Tags are indexed and fields aren't. Still starting with InfluxDB but that's how i understand it.
@AndreasSpiess4 жыл бұрын
I am not a specialist in InfluxDB and my data size is small. So I never investigated.
@byteborg5 жыл бұрын
@Andreas did you consider writing the name field not as a measurement but as a tag? This is achieved by writing not one object as msg.payload to the InfluxDB node, but writing a list [] of two objects, first one with the measurements, second one with the tags. If you set the name field as a tag, it should also show up automatically as a completion choice in Grafana.
@AndreasSpiess5 жыл бұрын
I did this after the video. It is easier for the selection in the dashboard. But during the take I did not know how to do it :-(
@g.s.33895 жыл бұрын
you'd better warn that grafana works on RPI3 only and not on previous raspberry boards.
@AndreasSpiess5 жыл бұрын
I did not know that.
@nicklaspersson46875 жыл бұрын
Thanks for warning Giuseppe Suanno. I was about to do this on a zero because i happen to have one available. Even if i suspected it might need a gruntier board like PI3. Why only on Pi3? Can you please elaborate or provide source?
@ajaybnl5 жыл бұрын
I am using it on orangepi lite.
@kiowablue28625 жыл бұрын
I've got it running on a RasPi 2, and have also ran it on the original 512MB RasPi Model B. Granted it's slower on the 1st generation Pi, but it runs with no unreasonable difference in speed on a Pi 2 compared to a Pi 3. (which I've also run it on)
@encephalophiliac2 жыл бұрын
Hi Andreas! Love this video, and thanks for putting together the image. I recently ordered a few Rasperry Pi Zero 2 Ws, and I'm trying to load the image you provided to one of them. When I load the flashed SD card and plug in the power, the device doesn't boot. No power light, no indication that it's connected to wifi (after adding wpa_supplicant.conf and SSH to the root directory). Is there any reason the image as provided wouldn't work on a raspi zero 2 W? Thanks again!
@AndreasSpiess2 жыл бұрын
I do not know if it works on the 2W. Two things to consider: 1. The memory of the 2W is very low for such applications. I would go for at least 2G 2. Maybe you watch the video about IOTstack. There you find another way of installing all the goodies...
@icecaper5 жыл бұрын
Great video again, Andreas and superb timing as I too have a project which requires this type of sensor recording/display. Have you used an IOT2020?
@AndreasSpiess5 жыл бұрын
No, I usually do not use such "professional" components like the IOT2020
@DaveBrown12 жыл бұрын
Excellent, very good.
@AndreasSpiess2 жыл бұрын
Thank you!
@daffdaffelito5 жыл бұрын
A verry good video! Thanks. I all ready got influxdb and node red on my server. But i never used the influxdb node in node red befor. It would be great if your node red flows where published somewhere!!
@AndreasSpiess5 жыл бұрын
They are: In the comments, you should find a link.
@daffdaffelito5 жыл бұрын
@@AndreasSpiess thanks! Looked a bit better now and found them.
@stephsoltesz67315 жыл бұрын
Good Day again ! Gosh I love your Video's and so timely at that ! After watching several of your vid's I'm struck by this particular one (#255) which answers many of my needs as to a project I want to get started on BUT some insight from you would be most helpful, I'm sure. The intended project, is to access my Solar System Components which include a Midnite Classic 200 MPPT Controller, a Samlex EVO-4024 Inverter/Charger using ModBus, pull stats & config info and provide a web based interface to view the info in Real Time and with full long term logging. In addition to that, I would like to setup a Weather Station (wind, rain etc) and a "Solar Monitor" of some sort that tells me the strength of the sun at points in the day (located with my fixed, ground mounted solar panels). Although I've never seen a "Solar Strength Sensor" as such, so this might be "interesting". Ultimately, I would like to be able to incorporate various Solar devices (brands) into a database which would allow people to be able to access the same info from their various devices. Something which can be distributed as a complete application or an image/iso/? I intend this project to be opensource / freeware in the Best Interests of everyone. Yup, I'm an RE Greeny ! LOL Any suggestions, thoughts or recommendations you have would be greatly appreciated ! Any recommended "Weather Stations" which can be monitored by a Pi would be helpful, as I'm sure some are better (easier) than others. Thanks in Advance for your thoughts & consideration Steve
@AndreasSpiess5 жыл бұрын
Maybe you slice this big elephant into smaller parts and start small. Then you can add all your bells and whistles as you go on. You will most probably make some mistakes and learn a lot on the way.
@stephsoltesz67315 жыл бұрын
@@@AndreasSpiess Very wise advice but being a IT Systems guy for 30 some years I have no fear of bigger projects with a longer view. ;-) Things have worked out wonderfully actually and again, thanks to your video's, you made my life so much easier so THANKS AGAIN ! I am now effectively communication with the Midnite Classic-200 Controller (Modbus via TCPIP) and writing data to Influx and then to Grafana. What a Great set of tools ! I have just received the confidential materials from Samlex (had to sign NDA's ;-( ) and purchased a USR-IOT RS485-> TCPIP converter (I maybe should have gotten an RS485 hat but I wanted some extra functionality is it handles RS232 as well_) so soon, I will be able to interface with the Inverter/Charger. Next stage to go with this is to setup a 12VDC -> 5VDC bus, from my Battery Bank to power the RasPi, USR Converter & a 9VDC line for the Router which would let them run outside of the solar system direct from the battery bank. Weather Station will follow in due course when I have time.
@tremartian38033 жыл бұрын
I suppose all the credentials should be changed to use the SD card image securely? What kind of changes should be done? Paswords for the Raspberry Pi, Node_red, influxDB and Grafana should be changed? It would be nice to have a guide how to do this and how changing the password of one app affects to another app.. in other words, how to keep the system working after passwords are changed.
@AndreasSpiess3 жыл бұрын
You can decide what you want to change. Because the server is behind my firewall I leave them as they are.
@leedevries56605 жыл бұрын
Great work again Andreas!
@AndreasSpiess5 жыл бұрын
Thank you!
@aliasname55182 жыл бұрын
Hello Andreas, does this work with an DHT22 Sensor or an Adafruit Sensirion SHT31-D? Where would I need to set the sensors up, in Node-Red? Thank you - great video!
@AndreasSpiess2 жыл бұрын
I usually do not connect sensors to the Raspberry because they have to be at different places. I use ESP32s and connect them with node-red via MQTT.
@Techonsapevole5 жыл бұрын
Thanks Andreas, did you consider to wrap this config in Docker/Docker Compose instead SD ? It would be easier to upgrade
@AndreasSpiess5 жыл бұрын
No, not for the moment. Maybe later...
@PhG19615 жыл бұрын
Waw, excellent work ! Heavy stuff aswell !
@AndreasSpiess5 жыл бұрын
The simple stuff is done by others ;-)
@LucasHartmann5 жыл бұрын
Playing with influx and grafana for computer (cpu ram ssd-cache and disk usage) and backup status monitoring. Got it working with docker containers after a few hours tinkering. Did not see the need for node red, though. Using curl for posting, and trying to get raspberry+Screenly working for display.
@AndreasSpiess5 жыл бұрын
I like node-red because I know it ;-) there are many other possibilities to bridge from MQTT into influxDB. Did you try docker to use GPIOs or Serial on the RPi?
@LucasHartmann5 жыл бұрын
@@AndreasSpiess I used docker for influx and grafana, not for the data collector /sensor program. You may be able to use gpio and serial inside a container by usingal a privileged container and mounting /sys and /dev/tty*** as volumes on it.
@nikolaimorozov63834 жыл бұрын
Many thanks for your video!
@AndreasSpiess4 жыл бұрын
You are welcome!
@tarmiziizzuddin3375 жыл бұрын
Thank you very much sir! Always love your video
@AndreasSpiess5 жыл бұрын
You are welcome!
@user-lj9uu9zb4w5 жыл бұрын
Thank you for the hundreds of really useful, interesting, up to date tutorials and info, you're pure gold! I have a question regarding the pi node-red server side, i need to have it in a distant location and feed it with 12V batteries (car battery) and let it run node-red influx db and grafana. What would be a good way to convert 12V down to 5V for the pi? Would one of these cheap buck converters do? What do you suggest? thank you in advance!
@AndreasSpiess5 жыл бұрын
A buck converter with at least 3A should do it. There are many projects around if you search with Google