AI in Home Assistant - A Complete Guide!

[@Don-wd2qn]

I would be interested in a video using Ollama on a separate server to control HA. Keep up the great work! Thanks!

[@SmartHomeJunkie]

I will try to set that up. First I need to set up a different server though.

[@jaylodewijckx2898]

@@SmartHomeJunkie i set it up a couple days ago and it took a long time to figure out how to set the varibels in windows so yea gread video idea

[@krzysztofkomornicki7007]

Definitely the best AI for Home Assistant video tutorial EVER!! Expecting more and more... Hungry of your next great ideas on how to use AI in a smart way

[@SmartHomeJunkie]

Thank you for your kind words

[@michaelsleen]

I’ve been enjoying LLM descriptions of who is at our front door for some time using LLM Vision. Excited about the potential of AI in home automations. Thanks, Ed.

[@SmartHomeJunkie]

Thank you for your response!

[@LyallJohnson]

Thank you for this tutorial. Helped me get started with AI responses

[@SmartHomeJunkie]

That's great to hear. You're welcome.

[@gerwingal4430]

Hey Ed thanks for this as always enlightening tutorial’s you create a lot of them have been very useful for me and now this one about AI will open a whole lot of new era of possibilities.

[@SmartHomeJunkie]

Yes! Enjoy your new automations!

[@Ukcavhead1]

Great tutorial and given me the idea to use it to monitor my driveway at night. Motion sensor on the Reolink doorbell captures anyone coming on there to interfere with something but also we get some false positives so I don't bother checking the Reolink playback each morning. By AI analysing if someone is in the snapshot and sending me a notification, I now know if there is a clip worth viewing. Thanks for the inspiration

[@SmartHomeJunkie]

Great idea

[@GilbertGagnon]

Wow I wasn't expecting to see these kind of ideas but amazing to see the possibilities and how easy it is, thank you very much!

[@SmartHomeJunkie]

Yes, you can do awesome things with it.

[@SteveJones172pilot]

Really good tutorial.. I would definitely like to see more, especially around the ollama. I am starting to think that while homeassistant doesn't need much hardware, all the supporting things to do what I want (ollama, frigate, etc) are going to take substantially more hardware!!

[@SmartHomeJunkie]

Yes, I think so too. I guess Ollama should run on a separate server. I will try to set that up!

[@KeithWeston]

Thanks! Helpful in the extreme.📸

[@SmartHomeJunkie]

Thank you for your support! 🙏🏻

[@AlexValtchev]

I am a big bug fan of yours. Lots of my HA automations and touch-ups are from youe videos. Please please make an LLM Vision tutorials with as much details as possible. Also, i have all Reolink cameras and doorbell, is there a way to hook them up to recognize faces and use AI to personalized responses based on the person they recognize? Perhaps with Blue Iris, Frigate or Motion Eye? I have never used any of them? Can you compare them as well? That will be awesome! 🤗😍

[@SmartHomeJunkie]

Thank you so much! I will see what I can do.

[@adfjasjhf]

I haven't seen LLM Vision covered anywhere yet. Would love to see it :)

[@SmartHomeJunkie]

Okay. Good to know.

[@scumhookleo]

Great vid, and a tutorial on Ollama install and config would be great!!!

[@SmartHomeJunkie]

Thank you so much.

[@MrCardeso]

Thanks! What a coincidence, I was recently thinking about having HA welcome me home with a funny message (but I was thinking about a static one). Also, I recently installed a Reolink doorbell!

[@SmartHomeJunkie]

Oh, it seems like your smart speaker is already eavesdropping on you. 🤪

[@MrCardeso]

@@SmartHomeJunkie Again?? I guess that's better than just giving me unprompted evil suggestions... again!

[@geoffreycoan]

Thanks Ed, interesting to see these real world examples of using Generative AI, the image analysis in particular is very clever. Not sure I really have a use case for AI yet, but good to see what’s possible

[@SmartHomeJunkie]

Thank you for your response!

[@hierichwerda]

Ich finde Deinen sozialen Gedanken bezüglich AI wirklich toll. Danke! Ich muss aber gleichermaßen sagen, dass ich ein wenig Angst habe vor diesen Unmengen an Daten die über die AI-Nutzer angesammelt werden. Die Menschheit wird gläsern. Olama würde mich sehr interessieren. Lokale Installationen sind immer mein Favorit.

[@SmartHomeJunkie]

Danke!

[@PatrickLangendoen]

Hey Ed, great video (as always)! I'll be experimenting with this. Also, as I'm a huge fan of keeping stuff local so I love to see a video about ollama as well 😎

[@SmartHomeJunkie]

I will look into Ollama. Let's see if it lives up to the expectations!

[@FlightdeckJohnny]

Thank you for another great video! Yes, I would like to see a test using a local ollama install and test the performance

[@SmartHomeJunkie]

That's great. I will look into it for sure.

[@dan-nutu]

+1 for this. It would be great if you could include some cheaper, older generation hardware, too, not just the latest CPUs with a hundred gigabytes of RAM or more, like many are showing. I'm not spending that much for a few tests with HA 😊

[@biosim5]

Bro you are always awesome! A tip: my welcome message is attach to my iPhone home location, with the iCloud HAS integration + if the main open door is opened just after the location change to Home.

[@SmartHomeJunkie]

Great idea!

[@sirmax91]

Why don't you track your phone using the router. Like using the nap tracker integration

[@SmartHomeJunkie]

I do this myself, but I wanted to make the tutorial easy so that it works without any additional hardware. Not everyone has a router that is supported.

[@sirmax91]

I get undefined error: 'response' is undefined​@@SmartHomeJunkie

[@SmartHomeJunkie]

Well then you must have forgotten to set the response variable.

[@Lonestar101]

Thanks for this tutorial! I'd like to learn more about the LLM's.

[@SmartHomeJunkie]

I only use Google Gemini and OpenAI.

[@Marqs_se]

Great tutorial. Keep them coming, please! 😊

[@SmartHomeJunkie]

I will!!!

[@frankr6053]

I love this. How can I push the images to an Amazon Echo?

[@gedgicat2063]

Great video, I can see how AI usage is a conflict and it's something i hadn't thought about, but your right it does seem to be the world is going 😢. Its great to see things from another point of view and how AI could be fun, and perhaps oneday help

[@SmartHomeJunkie]

I'm glad you agree.

[@ernstrietbergen1876]

Absolutely great, works like the best. Is it also possible to use home assistant cloud for speech?

[@SmartHomeJunkie]

Yes sure, use tts.cloud_say instead

[@SmartHomeJunkie]

I added a new blueprint to the blueprint pack that supports this!

[@ernstrietbergen1876]

@@SmartHomeJunkie i will check this out thanks🤗

[@resistme1]

I think you already had a tutorial covering the URI action since you don’t reference it for anyone wondering. What I do miss, however, as somebody stupid like myself, is some sense of how the variable works. It is hard to generalize it currently. For example, I would like to ingest the weather prediction into the LLM to suggest what appliances are viable to run and when during the day. I know I talk about two variables: the ingestion in the LLM (are variables a option there?) and the TTS action (response).

[@SmartHomeJunkie]

The service needs a variable to work with and I just use that. It's not that I define a custom variable or something.

[@RLockWee]

Are you able respond via the Nest Mini when it asks if it can do anything for you? Great video! Thanks so much!

[@SmartHomeJunkie]

I use the Voice PE to respond.

[@mattt9640]

Hi there, great video and yes please to a video about LLM video. This would be really good.

[@SmartHomeJunkie]

What do you mean by the LLM video?

[@drumslapper]

I can't agree more about AI and its influence on human nature. So far I have not allowed AI into my HA configuration but it is tempting for Voice Assist.

[@SmartHomeJunkie]

I'm glad I'm not the only one 🙏🏻

[@henriquer8453]

Thanks for the tutorial

[@SmartHomeJunkie]

Thank you for responding!

[@TheLawIX]

Constantly being around others can be as bad as isolation. Would love you to explore oppositely geared automations.

[@SmartHomeJunkie]

I'm sorry to hear that you feel bad about being around your family members.

[@TheLawIX]

@SmartHomeJunkie Weird response.

[@SmartHomeJunkie]

Okay, it was a serious response though.

[@metarichard]

Your HA technical videos are among the best on KZbin! Thank you! As and engineer I am fascinated with Ai but as a parent I am concerned about the impacts of Ai on my kids future. So keep doing what you are doing, you are one of the few who can see both sides!

[@SmartHomeJunkie]

Thank you. I am glad to read I'm not the only one!

[@stefan-lupo-pelzl]

An ollama video would be great!

[@SmartHomeJunkie]

I'll try to find time to investigate it.

[@RonaldvandeBeeten]

I use Google generative AI to analyse the cctv snapshots that are made when passing a tripwire. It tells me what generated the movement with the neccesery details like persons gender, look , clothing, type color and brand vehicle including licensplate numbers.

[@SmartHomeJunkie]

That's an awesome idea!

[@smART_attack]

I also want to write to an input helper the license plate. I made this through developer tools, but I didn't make it inside my automations.

[@trent529-d1q]

Great message and video! Do you happen to know where the "response.text" file/entity is? I'd love to have these messages appear on a dashboard, and while I can get them going as notifications on my phone I haven't been able to call them to a markdown card. The text input helper holds fewer characters too, so that isn't really an option

[@SmartHomeJunkie]

It's not an entity. It's just a variable that "lives" during the automation.

[@JbVest]

@SmartHomeJunkie I paid to get the data for one of these on your Kofi, I've done so many I forget which one it was lol😂 but, when I try to do the message portion after putting in the portion response.text with the brackets and run that section of the automation I get a pop-up that says. "value should be a string for dictionary value @ data['message']. Got None". Not sure where I'm going wrong I made sure the code was the same but still getting the error. ​

[@manfredzimmer1199]

Good Job Ed👍👍👍👍

[@SmartHomeJunkie]

Thanks 👍

[@AlexManMe]

This is going to help me immensely being visually impaired

[@SmartHomeJunkie]

Ah, I did not think about that. That's great.

[@anesiorsneto]

Thank you, Sir!

[@SmartHomeJunkie]

Thank you for responding!

[@pokemonfreaky100]

I’d like to see a video on how to install ollama!

[@SmartHomeJunkie]

Okay, noted!

[@fueralles]

Thanks for the video. You definitely know how to describe a complex automation. My only problem is that I can't figure out how to get the image send to my phone. It is saved in home assistant and the AI interpretation of the content is correct so it must be accessible but it never maid it to my android phone. Any advice?

[@SmartHomeJunkie]

No idea why you do not see an image on your phone. Did you use the correct url with /local/ in it?

[@davidhendrikx2553]

Thank you!

[@SmartHomeJunkie]

You're welcome!

[@erivaton]

Goede video. 👍 Zou ook graag weten hoe het werkt voor Grok.

[@SmartHomeJunkie]

Je kan LLM Vision daarvoor gebruiken.

[@Przedzik]

Gret, video !!! I am getting an error in tts that 'response' is undefined

[@SmartHomeJunkie]

Then you must have missed a step or you tried to test the action in the automation itself. It only works when the trigger is triggered.

[@Przedzik]

@@SmartHomeJunkie I appreciate your time for respond. Look like, this was just a typo, extra space I belive, all ok now thank You for another amazing tutorial

[@SmartHomeJunkie]

You are very welcome

[@feynthefallen]

I would be very interested to hear how the same is done using the openAI integration. The methods shown in this video unfortunately cannot be directly applied to that.

[@SmartHomeJunkie]

You can use LLM Vision for that.

[@giovanniongarato4302]

Great video! If i put the response variable as "response" in different automations, willl I create an issue or will Home Assistant understand that it has to display the last generated text from Gemini in the same specific notification? If 2 automations runs simultaneously and both asks Gemini to generate a text, with the same variable response, is it possible that the output will be incorrect?

[@SmartHomeJunkie]

The variable is automation specific, so it won't be an issue.

[@mvanimpe]

Thanks for the video Ed! Can you explain how you made the code to open the reolink app form within the notification? First time i see something like this... Thanks in advance!

[@SmartHomeJunkie]

Just use the url that I used and voilà

[@mvanimpe]

@@SmartHomeJunkie Thanks! It works!

[@gfdelbo]

@@SmartHomeJunkie and if I need to open a different application, how can I generate the link? I’m using Arlo camera

[@SmartHomeJunkie]

I don't know. I only know this for Reolink.

[@PeterPavlik72]

Hello & thank you for the interesting guide. I would like to know whether there is some option parameter for google tts to translate the response.text from english to some other language before sending to the media player?

[@SmartHomeJunkie]

Eh yes, I mention that in the video. You can choose what language you want when setting it up. Or maybe I don't understand your question.

[@PeterPavlik72]

​@@SmartHomeJunkieNot sure but I can setup language in HA only for Google TTS but not for AI service generating the response. Later I somehow managed to convince Gemini API to generate response in my local language but at first when testing your automation that seemed impossible so I wanted to ask.

[@RonaldChmara]

On AI assistant/assistance, I am still waiting until the reason for me using Home Assistant (local, not internet, control) has rich and tested /debugged (local, not internet), LLM model integration. It seems like we are *so close*, but right now, seems like there's still too much dependence on internet-connected OpenAI, Google AI (etc.) or really sketchy / hacky kludges to try and get a good, fast, local inference model.

[@lucabacchi7984]

Can you make a mix of 2 Blueprint ? :) When an event occurs, a photo is taken and AI analyzes according to what is requested, sending the content to an Echo chosen from the list. (example: when the doorbell rings, a photo is taken and AI announces via Alexa what he is wearing so we can understand who is at the door)

[@SmartHomeJunkie]

I can, but I’m not sure if it will play on an Echo.

[@lucabacchi7984]

@@SmartHomeJunkie ok i will wait your test :)

[@SmartHomeJunkie]

Haha, I don't have an Echo myself.

[@janneniemensivu475]

My daughter asked me to implement in Home Assistant a thing that she could at any time in our kitchen talk to Alexa and ask it to add an item to the shopping list. She said it would be really useful. Maybe, you could make a video of this using AI 0:39 ???

[@SmartHomeJunkie]

Yes, that actually works in Home Assistant using Assist with an LLM like Google Gecrashed AI. I use this myself. I might create a video about this.

[@leightonevans1071]

😊Ive got a bit of a daft question but how do i find out the image path of either my google door bell or reolink camera?

[@aktendully]

I'm curious how HA continues to integrate voice commands and AI further.

[@SmartHomeJunkie]

I use Google Generative AI for my voice assistant too and that works awesome.

[@1967_RS-SS]

Thanks!

[@SmartHomeJunkie]

Thanks for responding!

[@JiltedGeek]

Goeie video weer! dank ッ

[@SmartHomeJunkie]

Graag gedaan.

[@davidlucas1844]

Great video, thankyou. I have the notification come though but the images says ' Failed to load attachment, response status code was unacceptable :404' any ideas? thankyou

[@SmartHomeJunkie]

I guess you missed a step

[@TerryOReilly-yt6bh]

Hello, unrelated to this video but I'd like to ask a question. Do you have any videos on creating page tabs across the top of a page ? I think it was one of your video where I saw no side bar and page selections across the top.

[@SmartHomeJunkie]

Page tabs are default in HA dashboards. Not sure what you mean? Maybe you can watch my kiosk mode video.

[@djmattc1978]

Thanks for the video! Could you do a video on moving to zigbee2mqtt v2 from the latest v1? It has a lot of breaking changes and I cannot get my head around it.

[@SmartHomeJunkie]

I don't think I will. It's all in the release notes and the video will have a very short lifespan.

[@djmattc1978]

@ that’s fair enough. What about going from zigbee2mqtt to ZHA? That scares me 🥵

[@SmartHomeJunkie]

I wouldn't go from Z2M to ZHA.

[@josepz]

I need to capture the responses generated by google generative AI, when I write the prompt in the assist dialog window to send it to the speakers. I use the GGAI state change as the trigger event , and it works, but I don't know how to access to the generated text in this case... how can I do it?

[@nmeliasp]

curious how you found the URL for the reolink app that takes you to the specific camera

[@SmartHomeJunkie]

I saw it somewhere online. Not sure where though.

[@CPPRODUCTIONS1001]

Hmm, I tried getting this working on ring. The image and notification only comes through when I run the action manually but when motion is detected on ring, the event never triggers. It also seems to keep using the same snapshot, even when im standing in front of it, like it never overwrites the first image

[@SmartHomeJunkie]

You can add a random number to the image name to fix that.

[@CPPRODUCTIONS1001]

@@SmartHomeJunkie Ah, thank you. It seems to overwrite it but takes a long time to switch to the next image. I will keep playing around!

[@Fadukes]

Love the concept, however, today I installed Google Generative AI, and for the API was asked to “ Select a project from your existing Google Cloud projects”. I select “Generative Language Client”, but then no option appears in HA Developer Tools > Action > for “Google Generative AI”. I suspect something has changed regarding the Google API process?

[@SmartHomeJunkie]

No, nothing has changed. You probably activated this before in another project. Just remove it there and regenerate an api key.

[@Fadukes]

You were absolutely correct!! Don't understand why, but thanks.@@SmartHomeJunkie

[@SmartHomeJunkie]

Great that you solved it

[@ramirofdez.6630]

can i use other type speaker like apple homepod ?

[@SmartHomeJunkie]

As long as it is known as a media player in Home Assistant I guess you can.

[@arminsalchner]

Doras it also work with Alexa devices?

[@brianadriaanse3058]

Nice! ed is dit ook mogelijk met een iets echtere stem?

[@SmartHomeJunkie]

Je zou aan je vrouw kunnen vragen of ze jou op een grappige manier welkom heet. 😂😂😂 Maar nee, op dit moment is dit het. Het ligt ook een beetje aan de zin die wordt gegenereerd. Soms loopt deze wat vloeiender.

[@brianadriaanse3058]

@SmartHomeJunkie haha

[@danieleguastamacchia]

Any suggestion? Setup seems to be OK. Prompt text filled, and it works. I receive this error while using TTS "Error rendering data template: UndefinedError: 'response' is undefined". Seems like "response.text" does not exist.

[@SmartHomeJunkie]

You must have missed a step somewhere.

[@danieleguastamacchia]

@@SmartHomeJunkie solved. Basically the variable "response" (or whatever it is) is not available during a test of the automation but it works smooth during normal use or in a script.

[@michaelthompson657]

Anyone got the link to setup google Gemini? Can’t find it in the description and the blog version comes up with 404 and shows nothing

[@SmartHomeJunkie]

Oh, sorry, I forgot to add the links and publish the blog post. It's fixed now. The link is aistudio.google.com/

[@michaelthompson657]

@@SmartHomeJunkie thanks so much!

[@jonydupre]

Awesome video! Got the blueprint, but does somebody know how to make the message fully readable on Android? I cant expand the message like shown in the iPhone example. When I click on the message I'm redirected to HA or Reolink when clicking on the cam. But the message above the snapshot is cut off like this: here is a short description of a person...." Someone figured out how to show the full message? On Android I set the HA app to show details in notifications. Thanks!

[@SmartHomeJunkie]

On my iPhone I long press on the text to see the whole text.

[@jonydupre]

@SmartHomeJunkie hmm on Android this opens up the notification settings. Maybe someone else has a tip. :) thanks for the reply!

[@SmartHomeJunkie]

I'm curious as well.

[@jonydupre]

@@SmartHomeJunkie also in the notification I see an old snapshot. But when i look in the folder the old image is overwritten by the new one. Even when i run the automation again the old image is shown even though its no longer stored in the folder. Do you have any idea what could cause this?

[@SmartHomeJunkie]

I'm sorry but I don't know the solution to this.

[@MarcBoussen]

I get this error trying to save the automation (blueprint): Message malformed: extra keys not allowed @ data['triggers'] ? What am I doing wrong?

[@SmartHomeJunkie]

Someone else mentioned this as well. What's your HA version?

[@MarcBoussen]

@@SmartHomeJunkie 2024.9.2

[@SmartHomeJunkie]

You need at least version 2024.10 for it to work. I created an update of the Blueprints that now checks this version.

[@MarcBoussen]

thanks, I will try

[@janboerenjr]

Vooralsnog laat ik AI heel ver links liggen. Ik kijk er ouderwets tegenaan en heb nog geen echt nut gezien. De reclames can Gemini hebben geen verschil daarin gemaakt.

[@SmartHomeJunkie]

Verstandig!

[@benedictsmith2415]

Why does the voice over speaker response always start with "text" and "end" - sorta kills the immersion....

[@SmartHomeJunkie]

Because you didn't use the correct variable that ends with ".text"

[@benedictsmith2415]

@@SmartHomeJunkie Correct! You still have to steer it away from being too formal though e.g. Randomly pick from the following list of topics and then say something thought provoking in just two or three sentences without explicitly mentioning the topic you have chosen. The nature of consciousness, artificial intelligence.......

[@SmartHomeJunkie]

I'm concerned about AI. What's your opinion about AI? Let me know in the comments!

[@dan-nutu]

The cat is out of the bag so soon it'll be either sink or swim, for each of us, whether you like it or not. Also, whoever has the resources needed to train the models has an insane amount of unseen influence. I believe many many people will rely on AI without taking this into account.

[@MrWizardGG]

O boy yea, and I just got hired as an AI Solutions Developer

[@michaelthompson657]

Since when was Google Gemini free? I thought it was paid like OpenAI 🤔

[@SmartHomeJunkie]

It is free according to my Google AI Studio dashboard 🤷🏼‍♂️

[@michaelthompson657]

@ I had no idea! Do you think it’s worth signing up before it’s not free? Even if I might not use it?

[@SmartHomeJunkie]

I don't know really. It doesn't hurt I guess.

[@michaelthompson657]

@ no problem, thanks!

[@MrWizardGG]

Its free for me too

[@h3ld3rk1d]

Hello, do you know the way to send the response and the image to telegram chat? Thnkx

[@SmartHomeJunkie]

Currently I don't. But I know it is possible.

[@DonClassifieds]

Message malformed: extra keys not allowed @ data['triggers'] Ai Blueprint :(

[@SmartHomeJunkie]

I think you might have entered something wrong info in the blueprint. Which one of the two gives you this error and what data did you enter?

[@stiwie1]

Link google gemeni ?!?

[@SmartHomeJunkie]

Oh, sorry, I forgot to add the links and publish the blog post. It's fixed now. The link is aistudio.google.com/

[@bazcurtis178]

I have always loved technology, but it seems to me, in a lot of cases, it starts off with great positives, but soon it is abused for others gain. The cons soon outweigh the pros.

[@SmartHomeJunkie]

Exactly my thoughts. 🙏🏻

[@dan-nutu]

Same since our ancestors picked up rocks and clubs... the first to do this gains an advantage

[@AdventureRachel]

Aww I'm gutted i can't do this without shelling out even more money😢

[@SmartHomeJunkie]

Why? You can just follow the instructions in the video for free. 🤷🏼‍♂️

[@AdventureRachel]

@SmartHomeJunkie I don't have a VM machine or home assistant hardware

[@SmartHomeJunkie]

Ah! That was the meaning of your comment. Fair enough. 😊

[@Rndpntkrspnt]

Why would you let your phone (,phone means everybody) track you

[@SmartHomeJunkie]

What do you mean by "Everybody" tracks you? You make use of the HA companion app that detects your location.