Local Voice Assistant: Using your Cameras & Speakers in HA

In this tutorial, we’ll explore how to set up a local voice assistant powered by Assist using your smart cameras and speakers within Home Assistant (HA).
Even if you don't have any cameras on HA, we will show you how to test it with any Android Phone.
Update: Doesn't work with the Echo Players since they don't support local audio streaming
FixtSE Web: fixtse.com/blo...
Stream Assist Github Page: github.com/Ale...
00:00 Prerequisites
00:47 Install it on the Home Assistant Server
01:11 Install it on a Different Computer
02:21 Wyoming Integration
03:14 Configure Assistant
04:22 Stream Assist
05:07 Configure Stream Assist
05:35 Wake Word Detection Beep
07:28 Android IP Camera App
09:32 Demo
If you like my work, please consider supporting me on Ko-fi! ☕🎉: ko-fi.com/fixtse
Patreon: / fixtse
or Join this channel to get access to perks:
/ @fixtse.
You can find me on:
Web: fixtse.com/
Instagram: / fixtse
Hope this was useful and if you have any questions, write me a comment below
Thank you for watching (～￣▽￣)～

[@donnyf12]

Keep these Assist videos coming! You just solved a major problem I was having getting wake word working on Android. This is absolutely fantastic! Thank you!

[@fixtse.]

Glad it helped! 🙌

[@Mario-wu299]

FYI You can use Kodi on the phone to give feedback, sweet!

[@MrDenisJoshua]

Do you know if I can use one of the Google Nest Mini for this case ? Thanks for the video

[@elisalant]

Brilliant❤ Many thanks

[@ThomasBrasser1]

This is exactly my setup + Fully Kiosk as the media_player. Bonus: run rtsp stream through frigate and automate FKB (screen/screensaver) + StreamAssist based on motion/person/speech detected (Y)

[@emilianoruiz7681]

i cannot get this to work with my google home mini (1st gen)

[@meaningone]

thanks for the video. I've got it running using a cheap C100 tapo camera (I had to use it's RTSP stream directly as it was not discovered by the TAPO integration for whatever reason). I cannot say the result it great thoigh. the wakeword is detected from time to time only, and no sound is coming out of any of my media_player.x devices. any advice ?

[@fixtse.]

Activate the debug mode for the assist pipeline, grab the recording from one of the times that it didn't picked up the sound, see if there is a problem with the recording quality, it should sound about the same as the video the camera can record by itself.

[@antoniomazabandes5419]

Great, you're a crack. You should get last year's Emi award and this year's award. Just a little thing, those of us who don't speak or understand English fluently, would appreciate it if you wouldn't be so fast. Thank you, thank you very much

[@fixtse.]

Jajaja thank you, believe me I'm working on that 😅, I hope to keep improving over time 🤞🙌

[@FroMan753]

Does this work with the new FullyKiosk camera entity that was just introduced in the 2024.7 update? If so, we finally have a workaround (an easy one at that) to getting wakeword assist on wall tablets

[@fixtse.]

Let's find out.

[@mikesmods317]

Awesome job!

[@lsattle]

I followed this great video trying to use the 'ip camera' app on an android phone. I got the system to respond to my wake word by seeing the status changes. However, I have no speaker installed like you have in the video and was hoping the speaker in the phone would do the responding. Your video has the sound coming out of a speaker. When you were setting up the use of your camera you selected a camera and media buzzwords were involved. I have no media buzzwords and the home assistant web site is not of any assistance that I can find. Note, I have this running on an old intel i3 laptop that runs fine with piper and whisper running locally. Can I use the android phone as a speaker somehow to at least get moving better with the app before investing in more hardware?

[@fixtse.]

I'll put together a video, just be patience.

[@Mario-wu299]

@@fixtse. I think this would be useful to all of us if you can pull this off, having the android be able to play the beep/ give feedback would be awesome!

[@Sing00525]

Can I connect a USB mic to my HA RPi for the audio input without using an IPcamera?

[@LordFortesQ]

It would be interesting to see if a feature can be created that plays the latest news from specific sources such as CNN and others. That's one of the main benefits of using the Google Assistant.

[@JasonKnightDJJasonK]

Thanks for the great video. I have it up and running now. Is there anyway to get extended conversation working? How can I auto trigger wake, so that it listens for my response?

[@thesecretreviewer8242]

i use my Reolink cameras as speakers and mics. works well.

[@FroMan753]

How do you use them as the speakers? I don't see how to have them in home assistant as a media player

[@KrisHoobergs]

Hi I have the same problem as @FroMan753 I can use my Reo as a mic but don't see it as a media player.

[@goodcitizen4587]

Really cool! Can you do a vid on examples of music/media streamers that are good to use with HA. Thanks!

[@virenderbhanot]

How do we configure the actions each command performs when we utter ì to assist. Thanks for the great tutorial.

[@fixtse.]

I'll do a follow-up video with more usage examples, to trigger automations and scripts. It is also possible to configure your own trigger sentences.

[@GoranSutic-hd2sz]

Great video. Keep up the good work. I wonder if any of the Xiaomi Smart speakers or similar devices can be used as microphones?

[@fixtse.]

Hi, thank you, I don't think so. Unless you can get an RTSP, HTTP or RTMP stream with audio into Home Assistant, it's not going to be possible. For example, to get the RTSP Stream for the Nest Hub Max into Home Assistant, you need the nest integration (it requires a $5 fee if i'm correct), and even after that, you can only get Video, not audio. So it wouldn't work with this integration. I'm doing some research about this, so expect a video in the near future 😁

[@AJD_YE]

Great. Can I use Amazon Echo Dots as microphone and speaker and media player?

[@fixtse.]

No, one viewer just confirmed that it didn't work with Alexa. It's because of the way that the Alexa integration had to be implemented (it's a cloud integration, not a local one). Echo devices don't support local playback

[@IoSonoAndreaZ]

i can't manage to make the STT start media work, despite i'm following the same steps. I can manually play the mp3 from the media tab on any device but it never plays when I use it for voice commands

[@fixtse.]

you might be willing to give my azure tts stt video a chance, to verify if it's a problem with the integration or with the service

[@IoSonoAndreaZ]

@@fixtse. solved the issue, the path is a bit different when using docker

[@IoSonoAndreaZ]

@@fixtse. problem solved. For my setup (docker core) the correct path was media-source://media_source/media/beep.mp3

[@tduf1]

Great video 😊 how to do that with an ESP32-S3-BOX-3 ? Thks.

[@fixtse.]

It should be easier, but not with this method. You need to go the esphome route to get the ESP32-S3-BOX-3 working with Home Assistant. I don't have the device, but I've seen that it even supports on-device hot word detection.

[@CleiPereira]

hello, how to add the beep path that is inside the www folder?

[@MatthewN8OHU]

Running a Local LLM is the best way to replicate the capabilities of a Google Home Mini or Amazon Echo without having to have an internet connection. It's like having J.A.R.V.I.S. from Iron Man without all the effort that Tony Stark put into creating and training him.

[@fixtse.]

Yes, I think I should upload a short showing some wild answers from the AI I got, some of them are so bad that they are good 😂😂

[@shaweewishissime6718]

Hi, tell me how did add LLMs in Conversation Agent, and how you have so much voices?

[@fixtse.]

Check out my other videos for the LLMs part, for the voices I actually show it here, using a docker piper installation

[@shaweewishissime6718]

@@fixtse. Sorry my friend, I saw that after my comment. :/ Merci mon ami ;)

[@michaelthompson657]

For this could I just use a microphone and a separate speaker? I have a spare pi 3b I could put HA satellite on and use a pi hat then I have a Sonos one. Could they be used together like this instead of a camera?

[@fixtse.]

Not sure, I would have to check how Wyoming Satellites work, but, let's say it is possible, It will require adding that as an option to the integration, so it can redirect the output to the Sonos speaker. I think you should add this as a question on the GitHub page of the project (on the Issues tab, since the repo doesn't have Discussions Activated), so AlexxIT can give it a look

[@michaelthompson657]

@@fixtse. thanks I’ll have a look

[@Mr-Coke]

I love youre Videos. That was what I waiting for. Great. Thank you. Do you have a solution for only Integrate microphones instead of a mic from a camera? Because mics a easier to Place. Thank you Feedback: - it didnt worked with alexa speaker. didnt get sound from. - unfortunately i cant run it on my walldashboard with the android up because then the camera is blogged and fully cant use it.

[@fixtse.]

Hey thank you for the feedback 🙌🙌🙌 I was wondering if it worked on Alexa 😕, I was hoping it did since it's just playing a audio file, I'll update the description. Let me see if I can find a way to use just the mic in the future, you'll never know what can be done with some clever code 🙌🙌🙌

[@Mr-Coke]

@@fixtse. I integrated now an s3 Box lite and combined it with my echo speaker. Unfortunately the Internal speaker speaks too, Do you know how I can Mute or deactivate this? Next step esp32 with mic and combined with the echo.

[@pjuhl2313]

What cameras have you tried and found work? My one camera in my living room has a mic but I could not get the SST to become active.

[@fixtse.]

Any RTSP, OVNIF or RTMP should work, the integration handles the transcoding of the audio source into something suitable for STT automatically. I use YiDome Cameras with the roleoroleo firmware to support the RTSP protocol, but any camera should work.

[@pjuhl2313]

@@fixtse. Any camera with a microphone, correct?

[@fixtse.]

@@pjuhl2313 yes, as long as the firmware supports audio over RTSP, OVNIF or RTMP. That is up to the manufacturer

[@pjuhl2313]

@@fixtse. What integration are you using for your cameras in HA. I'm using Frigate and wondering if I need to allow audio in the config for it to transfer over to HA

[@fixtse.]

@@pjuhl2313 I use frigate too, but I'm using the YiHack Integration Camera, instead of going through frigate. If you want to use it with frigate, I think, as you said, that you need add audio support on your frigate config files, there are examples on frigates the documentation, I put a link on my frigate article on my website if I recall correctly.

[@jeroenbrouwers3976]

Do you need a camera? Or can I use a Google mini as mic input?

[@jlpnea]

I have the same question

[@fixtse.]

Yes, you need the camera. Google doesn't expose access to the mic on its devices, so there is no way for home assistant to access that stream.

[@user-xc5zw2cg1u]

I JUST was wondering why my cameras have so many sensors and I can’t use them to do more. This is perfect! If only I could use my camera speakers as a media player for something like this (not music obvs…).

[@fixtse.]

It is possible, if your camera support two way audio you can use WebRTC ( github.com/AlexxIT/WebRTC#stream-to-camera ) Camera custom integration to add it as Media player.

[@user-xc5zw2cg1u]

@@fixtse.Wow thanks! Your videos have stuff I don't see anywhere else. I appreciate you posting the steps so precisely.

[@user-xc5zw2cg1u]

@@fixtse. I have read through the WebRTC information and googled a lot of discussions. I can't find much info on the new Stream to Camera option. May make a good video. I can't get it working and see a lot of people struggling also.

[@pjllb]

This solution took my Raspberry Pi hos CPU utilization from ~4 percent to well over 20 percent. The solution's functionality is good but not acceptable overall based on the system impact.

[@fixtse.]

I think that your point is valid, I'll try to include more related information in the future. But yes, that is normal, specially if you are running all the add-ons in one device, or use ffmpeg to transcode the stream to something that home assistant can process, it is to be expected.

[@rastkokostic]

Hi, does home assitant green has enough juice to run this pipeline efficiently and fast engough?

[@fixtse.]

No, unless you offload at least whisper and piper to a more powerful machine.

[@pablopnJCPN]

I understand that a Google Nest Mini 2 generation can be used, what happens to me is that when I put several, it only responds and listens to one. does it happen to anyone?

[@fixtse.]

Interesting, I haven't test this scenario yet, could take a while, but i'll add it to my list and get back to you when I have an answer.

[@pablopnJCPN]

@@fixtse. Thanks for your time and videos!

[@redaste]

Amazing effort! Does it support Google home mini?

[@fixtse.]

Yes it does

[@laukhengsoon]

Thanks for sharing, what mic do you use in this video? TIA

[@fixtse.]

The mic on the YiDome Camera, I use the yi-hack firmware to get an RSTP stream from the camera with audio support.

[@laukhengsoon]

Great, thanks for sharing.😊

[@AJD_YE]

Can I implement this pipeline: Amazon Echo Dot > Home Assistant > Custom Wake Word > Fast GPT > Home Assistant> Device Actions > Confirmation via Amazon Echo Dot?

[@fixtse.]

No.

[@AJD_YE]

​@@fixtse.Thank you very much for your answer. What a shame the pipeline does work with Amazon Echo devices. From my point of view Amazon Echo devices are the best smart speaker. Google seams to have stopped development of Google/Nest devices and the quality of ESP32 Devices isn't as good as Amazon Echo devices. I have quite a lot of rooms. What is a reliable hardware reference?

[@raimundasskurdenis]

is posible with Homepod? Or Sonos One?

[@antoniotomeo4590]

Great. Can I use Google Home Mini as microphone and speaker and media player?

[@fixtse.]

no

[@MrDenisJoshua]

@@fixtse. In the comment at the question " Does it support Google home mini?" your answer was "yes"... and here the answer is "no"... I just don't want to initiate all the installations if google mini is not work So witch one of the answer is the true please ?

[@Andy15792]

@@MrDenisJoshua It is possible to use as the speaker but not as the mic. I am trying to figure that out too, as right now from the integration only RTSP/HTTP/RTMP protocol works for mic. So anything that can stream the speech via that should work. Wondering if I can have pi hole running with mic that can do that, will that work? cc @fixtse

[@MrDenisJoshua]

@@Andy15792 I'm newbee too... sincerely I don't know :-)

[@elisalant]

Anyone have any idea as to how to add a Google speaker to HA?

[@fixtse.]

They should work out of the box, as long as they are on the same local network.

[@victorfsaaa]

Is it possible to make it without a wake word?

[@fixtse.]

Yes, V1 used to work like that, I never used it, but i guess you just need to call a service to trigger the voice assistant process.

[@rastkokostic]

How to #Automate Script Execution at logon? Where to copy provided code?

[@fixtse.]

fixtse.com/blog/ollama-home-assistant#automate-script-execution-at-logon

[@rastkokostic]

@@fixtse. But where to copy provided code?

[@sayam93]

@fixtSE Would this work for a google nest cam battery 2nd generation (given the limitations of how sdm api works)? If not, could I use a Wyoming Satellite (kzbin.info/www/bejne/m4WumJZmjql2rac) as an audio input? P.S. I have openwakeword running on the satellite device and am using Home Assistant Cloud for STT and TTS.

[@fixtse.]

Hi, I can't answer that question since I don't have the device myself, but you can, If you can get a RTSP, HTTP or RTMP stream with audio from your camera into Home Assistant, yes, you can, if not, it's not possible. Right now, Wyoming satellite as an audio source is not supported, but you can add a feature request for it on the project GitHub page. I'm sure that if it's possible AlexxIT will consider adding it. (It kind of goes out of the main scope of the project tho, so be respectful if he says that he is not planning on supporting that feature)

[@youyuberh]

will my google home still work with "ok google" ?

[@fixtse.]

Yes.

[@AviDarks]

cool. can i use different voice assist not english?

[@fixtse.]

Yes, it is available in different languages, just keep in mind that the accuracy of the detection will vary.