Local Voice Assistant: Using your Cameras & Speakers in HA

[@donnyf12]

Keep these Assist videos coming! You just solved a major problem I was having getting wake word working on Android. This is absolutely fantastic! Thank you!

[@fixtse.]

Glad it helped! 🙌

[@michaelmcalceet]

It would be interesting to see if a feature can be created that plays the latest news from specific sources such as CNN and others. That's one of the main benefits of using the Google Assistant.

[@ThomasBrasser1]

This is exactly my setup + Fully Kiosk as the media_player. Bonus: run rtsp stream through frigate and automate FKB (screen/screensaver) + StreamAssist based on motion/person/speech detected (Y)

[@alexwang8075]

What is Fully Kiosk? In HACS only find a "kiosk mode" which didn't provide media player

[@ThomasBrasser1]

@@alexwang8075 android app

[@MrDenisJoshua]

Do you know if I can use one of the Google Nest Mini for this case ? Thanks for the video

[@elisalant]

Brilliant❤ Many thanks

[@robertperic6505]

Hi. You make extraordinary tutorials for Home assistant users. I would like to know how to set the tablet, where I have the Home Assistant companion APP and Google voice access installed, so that the tablet, in addition to displaying the user interface, also functions as a voice assistant? If you have any ideas, I would be happy to see a video on this topic.

[@antoniomazabandes5419]

Great, you're a crack. You should get last year's Emi award and this year's award. Just a little thing, those of us who don't speak or understand English fluently, would appreciate it if you wouldn't be so fast. Thank you, thank you very much

[@fixtse.]

Jajaja thank you, believe me I'm working on that 😅, I hope to keep improving over time 🤞🙌

[@TrevorV]

You can always change the playback speed of the video to a slower amount :)

[@mikesmods317]

Awesome job!

[@thesecretreviewer8242]

i use my Reolink cameras as speakers and mics. works well.

[@FroMan753]

How do you use them as the speakers? I don't see how to have them in home assistant as a media player

[@KrisHoobergs]

Hi I have the same problem as @FroMan753 I can use my Reo as a mic but don't see it as a media player.

[@Mario-wu299]

FYI You can use Kodi on the phone to give feedback, sweet!

[@alexwang8075]

Tried not successful

[@Hasse-swe]

Hi. Yhanks for the video. At 3:54 when you are going to test the voice andget an error saying "Unable to load example. Response error: 500" you need to go to Setting -> System -> Network and add the adress you ar using to connect to home assistant to get it to work. For me i added my duckdns adress to get it to work. Then i cant get it to work with my Google Assistant at all.

[@goodcitizen4587]

Really cool! Can you do a vid on examples of music/media streamers that are good to use with HA. Thanks!

[@virenderbhanot]

How do we configure the actions each command performs when we utter ì to assist. Thanks for the great tutorial.

[@fixtse.]

I'll do a follow-up video with more usage examples, to trigger automations and scripts. It is also possible to configure your own trigger sentences.

[@AJD_YE]

Great. Can I use Amazon Echo Dots as microphone and speaker and media player?

[@fixtse.]

No, one viewer just confirmed that it didn't work with Alexa. It's because of the way that the Alexa integration had to be implemented (it's a cloud integration, not a local one). Echo devices don't support local playback

[@MatthewN8OHU]

Running a Local LLM is the best way to replicate the capabilities of a Google Home Mini or Amazon Echo without having to have an internet connection. It's like having J.A.R.V.I.S. from Iron Man without all the effort that Tony Stark put into creating and training him.

[@fixtse.]

Yes, I think I should upload a short showing some wild answers from the AI I got, some of them are so bad that they are good 😂😂

[@lsattle]

I followed this great video trying to use the 'ip camera' app on an android phone. I got the system to respond to my wake word by seeing the status changes. However, I have no speaker installed like you have in the video and was hoping the speaker in the phone would do the responding. Your video has the sound coming out of a speaker. When you were setting up the use of your camera you selected a camera and media buzzwords were involved. I have no media buzzwords and the home assistant web site is not of any assistance that I can find. Note, I have this running on an old intel i3 laptop that runs fine with piper and whisper running locally. Can I use the android phone as a speaker somehow to at least get moving better with the app before investing in more hardware?

[@fixtse.]

I'll put together a video, just be patience.

[@Mario-wu299]

@@fixtse. I think this would be useful to all of us if you can pull this off, having the android be able to play the beep/ give feedback would be awesome!

[@redaste]

Amazing effort! Does it support Google home mini?

[@fixtse.]

Yes it does

[@TrevorV]

I would love to buy some hardware to put around the house to do this. I want something that I can use as a voice assistant, but I also really want something that can play music from Music Assistant. Any suggestions?

[@fixtse.]

From what I've dug into, there isn't a definite alternative at the moment, but many people are working on it, and esphome is making solid progress to improve audio support, so the future its looking great.

[@thefunkygibbon]

this is great , I've got it now set up and using Google Gemini ... but I'm finding it hideously slow from the time I ask it to the time it reads the answer to me. if I use my little m5stack atom echo device which is listed under the "assist devices" I get replies almost immediately. using stream assist it's taking about 30 seconds .. also it seems that I have to ask the question immediately after saying ok nabu otherwise it gives up thinking I've not said anything within half a second.

[@fixtse.]

Lately, I’ve noticed slow response times with Gemini, even during direct assist chats. Initially, I suspected it was an issue with my ISP since it doesn’t occur consistently, with Ollama it works instantly on my setup. Regarding the “no words detected” error, I’ve experienced that as well. I assumed it was an assist-related issue, but after trying out the ReSpeaker Lite from Seeed Studio, I think is something with Strem Assist.

[@FroMan753]

Does this work with the new FullyKiosk camera entity that was just introduced in the 2024.7 update? If so, we finally have a workaround (an easy one at that) to getting wakeword assist on wall tablets

[@fixtse.]

Let's find out.

[@laukhengsoon]

Thanks for sharing, what mic do you use in this video? TIA

[@fixtse.]

The mic on the YiDome Camera, I use the yi-hack firmware to get an RSTP stream from the camera with audio support.

[@laukhengsoon]

Great, thanks for sharing.😊

[@SieciowyToster]

I have problem with instalation whisper. Everything looks correct, but the add-on does not start, the launch arrow spins endlessly and that's it.

[@fixtse.]

Check the logs tab and see if you get any error there. On what hardware are you trying to run it? (full specs)

[@JasonKnightDJJasonK]

Thanks for the great video. I have it up and running now. Is there anyway to get extended conversation working? How can I auto trigger wake, so that it listens for my response?

[@meaningone]

thanks for the video. I've got it running using a cheap C100 tapo camera (I had to use it's RTSP stream directly as it was not discovered by the TAPO integration for whatever reason). I cannot say the result it great thoigh. the wakeword is detected from time to time only, and no sound is coming out of any of my media_player.x devices. any advice ?

[@fixtse.]

Activate the debug mode for the assist pipeline, grab the recording from one of the times that it didn't picked up the sound, see if there is a problem with the recording quality, it should sound about the same as the video the camera can record by itself.

[@antoniotomeo4590]

Great. Can I use Google Home Mini as microphone and speaker and media player?

[@fixtse.]

no

[@MrDenisJoshua]

@@fixtse. In the comment at the question " Does it support Google home mini?" your answer was "yes"... and here the answer is "no"... I just don't want to initiate all the installations if google mini is not work So witch one of the answer is the true please ?

[@Andy15792]

@@MrDenisJoshua It is possible to use as the speaker but not as the mic. I am trying to figure that out too, as right now from the integration only RTSP/HTTP/RTMP protocol works for mic. So anything that can stream the speech via that should work. Wondering if I can have pi hole running with mic that can do that, will that work? cc @fixtse

[@MrDenisJoshua]

@@Andy15792 I'm newbee too... sincerely I don't know :-)

[@RafaelAbrantes]

My Whisper doesnt start. I get the error: "Your CPU does not support Advanced Vector Extensions (AVX)." Any tips on how to resolve this? Thanks in advance.

[@Mr-Coke]

Do you know a way how i can integrade the speaker from the camera into HA Then i could talk to the assistant everywhere in my garden. There i didnt have an speaker, but i have speaker in the cameras.

[@Mr-Coke]

okay i have it. my camera speaker is integrated as a normal Media Player. And TTS with the camera speaker works now. But the quality is not so good. Needs a little bit finetuning i think.

[@tduf1]

Great video 😊 how to do that with an ESP32-S3-BOX-3 ? Thks.

[@fixtse.]

It should be easier, but not with this method. You need to go the esphome route to get the ESP32-S3-BOX-3 working with Home Assistant. I don't have the device, but I've seen that it even supports on-device hot word detection.

[@AubreyNelsonPage]

I JUST was wondering why my cameras have so many sensors and I can’t use them to do more. This is perfect! If only I could use my camera speakers as a media player for something like this (not music obvs…).

[@fixtse.]

It is possible, if your camera support two way audio you can use WebRTC ( github.com/AlexxIT/WebRTC#stream-to-camera ) Camera custom integration to add it as Media player.

[@AubreyNelsonPage]

@@fixtse.Wow thanks! Your videos have stuff I don't see anywhere else. I appreciate you posting the steps so precisely.

[@AubreyNelsonPage]

@@fixtse. I have read through the WebRTC information and googled a lot of discussions. I can't find much info on the new Stream to Camera option. May make a good video. I can't get it working and see a lot of people struggling also.

[@pjuhl2313]

What cameras have you tried and found work? My one camera in my living room has a mic but I could not get the SST to become active.

[@fixtse.]

Any RTSP, OVNIF or RTMP should work, the integration handles the transcoding of the audio source into something suitable for STT automatically. I use YiDome Cameras with the roleoroleo firmware to support the RTSP protocol, but any camera should work.

[@pjuhl2313]

@@fixtse. Any camera with a microphone, correct?

[@fixtse.]

@@pjuhl2313 yes, as long as the firmware supports audio over RTSP, OVNIF or RTMP. That is up to the manufacturer

[@pjuhl2313]

@@fixtse. What integration are you using for your cameras in HA. I'm using Frigate and wondering if I need to allow audio in the config for it to transfer over to HA

[@fixtse.]

@@pjuhl2313 I use frigate too, but I'm using the YiHack Integration Camera, instead of going through frigate. If you want to use it with frigate, I think, as you said, that you need add audio support on your frigate config files, there are examples on frigates the documentation, I put a link on my frigate article on my website if I recall correctly.

[@GoranSutic-hd2sz]

Great video. Keep up the good work. I wonder if any of the Xiaomi Smart speakers or similar devices can be used as microphones?

[@fixtse.]

Hi, thank you, I don't think so. Unless you can get an RTSP, HTTP or RTMP stream with audio into Home Assistant, it's not going to be possible. For example, to get the RTSP Stream for the Nest Hub Max into Home Assistant, you need the nest integration (it requires a $5 fee if i'm correct), and even after that, you can only get Video, not audio. So it wouldn't work with this integration. I'm doing some research about this, so expect a video in the near future 😁

[@rastkokostic]

Hi, does home assitant green has enough juice to run this pipeline efficiently and fast engough?

[@fixtse.]

No, unless you offload at least whisper and piper to a more powerful machine.

[@jeroenbrouwers3976]

Do you need a camera? Or can I use a Google mini as mic input?

[@jlpnea]

I have the same question

[@fixtse.]

Yes, you need the camera. Google doesn't expose access to the mic on its devices, so there is no way for home assistant to access that stream.

[@shaweewishissime6718]

Hi, tell me how did add LLMs in Conversation Agent, and how you have so much voices?

[@fixtse.]

Check out my other videos for the LLMs part, for the voices I actually show it here, using a docker piper installation

[@shaweewishissime6718]

@@fixtse. Sorry my friend, I saw that after my comment. :/ Merci mon ami ;)

[@gguy156]

You are odd fella but I must admit you steamed a good vid

[@emilianoruiz7681]

i cannot get this to work with my google home mini (1st gen)

[@CleiPereira]

hello, how to add the beep path that is inside the www folder?

[@AJD_YE]

Can I implement this pipeline: Amazon Echo Dot > Home Assistant > Custom Wake Word > Fast GPT > Home Assistant> Device Actions > Confirmation via Amazon Echo Dot?

[@fixtse.]

No.

[@AJD_YE]

​@@fixtse.Thank you very much for your answer. What a shame the pipeline does work with Amazon Echo devices. From my point of view Amazon Echo devices are the best smart speaker. Google seams to have stopped development of Google/Nest devices and the quality of ESP32 Devices isn't as good as Amazon Echo devices. I have quite a lot of rooms. What is a reliable hardware reference?

[@Sing00525]

Can I connect a USB mic to my HA RPi for the audio input without using an IPcamera?

[@alexwang8075]

Yes you can by using FFMPEG to capture the Mic input as a RTSP flow

[@alexwang8075]

Yes you can by using FFMPEG to capture the Mic input as a RTSP flow

[@IoSonoAndreaZ]

i can't manage to make the STT start media work, despite i'm following the same steps. I can manually play the mp3 from the media tab on any device but it never plays when I use it for voice commands

[@fixtse.]

you might be willing to give my azure tts stt video a chance, to verify if it's a problem with the integration or with the service

[@IoSonoAndreaZ]

@@fixtse. solved the issue, the path is a bit different when using docker

[@IoSonoAndreaZ]

@@fixtse. problem solved. For my setup (docker core) the correct path was media-source://media_source/media/beep.mp3

[@Mr-Coke]

I love youre Videos. That was what I waiting for. Great. Thank you. Do you have a solution for only Integrate microphones instead of a mic from a camera? Because mics a easier to Place. Thank you Feedback: - it didnt worked with alexa speaker. didnt get sound from. - unfortunately i cant run it on my walldashboard with the android up because then the camera is blogged and fully cant use it.

[@fixtse.]

Hey thank you for the feedback 🙌🙌🙌 I was wondering if it worked on Alexa 😕, I was hoping it did since it's just playing a audio file, I'll update the description. Let me see if I can find a way to use just the mic in the future, you'll never know what can be done with some clever code 🙌🙌🙌

[@Mr-Coke]

@@fixtse. I integrated now an s3 Box lite and combined it with my echo speaker. Unfortunately the Internal speaker speaks too, Do you know how I can Mute or deactivate this? Next step esp32 with mic and combined with the echo.

[@Mr-Coke]

@@fixtse. i now using an esp32 as mic and connected this with my alexa speaker. with a own script i now can doing reall conversations instead of needing the wakeword every time. HA is sending a mp3 file to alexa. This file i open and read the duration of the file and put a few seconds on it. After the time is over the mics starting again from alone without a wakeword. But only if you want this.

[@fixtse.]

@@Mr-Coke 🙌🙌🙌🙌🙌🙌🙌🙌 that's great! and with alexa, who would have thought.

[@Mr-Coke]

@@fixtse. I want to send you a link with the Code but yt delete it

[@raimundasskurdenis]

is posible with Homepod? Or Sonos One?

[@michaelthompson657]

For this could I just use a microphone and a separate speaker? I have a spare pi 3b I could put HA satellite on and use a pi hat then I have a Sonos one. Could they be used together like this instead of a camera?

[@fixtse.]

Not sure, I would have to check how Wyoming Satellites work, but, let's say it is possible, It will require adding that as an option to the integration, so it can redirect the output to the Sonos speaker. I think you should add this as a question on the GitHub page of the project (on the Issues tab, since the repo doesn't have Discussions Activated), so AlexxIT can give it a look

[@michaelthompson657]

@@fixtse. thanks I’ll have a look

[@alexwang8075]

Yes you can. Use FFMPEG to capture the Mic input as a RTSP stream

[@michaelthompson657]

@@alexwang8075 could you elaborate please? I’ve never used ffmpeg or rtsp and I thought they were for cameras?

[@giordanogobbo]

did you succeed? i've tried many option but sonos does not play the sound or the tts

[@pablopnJCPN]

I understand that a Google Nest Mini 2 generation can be used, what happens to me is that when I put several, it only responds and listens to one. does it happen to anyone?

[@fixtse.]

Interesting, I haven't test this scenario yet, could take a while, but i'll add it to my list and get back to you when I have an answer.

[@pablopnJCPN]

@@fixtse. Thanks for your time and videos!

[@AviDarks]

cool. can i use different voice assist not english?

[@fixtse.]

Yes, it is available in different languages, just keep in mind that the accuracy of the detection will vary.

[@sansopheakh6943]

How to make continue conversation without saying Wake Word again?

[@fixtse.]

Not possible yet.

[@victorfsaaa]

Is it possible to make it without a wake word?

[@fixtse.]

Yes, V1 used to work like that, I never used it, but i guess you just need to call a service to trigger the voice assistant process.

[@rastkokostic]

How to #Automate Script Execution at logon? Where to copy provided code?

[@fixtse.]

fixtse.com/blog/ollama-home-assistant#automate-script-execution-at-logon

[@rastkokostic]

@@fixtse. But where to copy provided code?

[@pjllb]

This solution took my Raspberry Pi hos CPU utilization from ~4 percent to well over 20 percent. The solution's functionality is good but not acceptable overall based on the system impact.

[@fixtse.]

I think that your point is valid, I'll try to include more related information in the future. But yes, that is normal, specially if you are running all the add-ons in one device, or use ffmpeg to transcode the stream to something that home assistant can process, it is to be expected.

[@youyuberh]

will my google home still work with "ok google" ?

[@fixtse.]

Yes.

[@elisalant]

Anyone have any idea as to how to add a Google speaker to HA?

[@fixtse.]

They should work out of the box, as long as they are on the same local network.