LOCAL Voice and Hearing for your Home Assistant Assist!!!

[@big-ted4065]

Thanks for an in detail video, won't be using it until a wake word is available for a handsfree experience

[@BeardedTinker]

Yeah, that's the "only" missing part now. Wondering how they'll overcome issue with custom HW for it.

[@Claudiu.]

Very comprehensive and enough details to understand these tools. Thank you!

[@BeardedTinker]

Thank you - still testing some bits but looks very promising for 2nd iteration. Can't wait to see what will be next.

[@synesthesiam]

Mike "The Voice" here from Nabu Casa. Excellent video! Any chance you'd be interested in helping me create a Croatian Piper voice?

[@BeardedTinker]

Thank you :) Sure, why not - you can ping me on Discord, or better, I'll ping you there.

[@texanallday]

This is awesome!

[@gbadishomeassistant748]

The best video far away than others. Thanks from Greece.

[@BeardedTinker]

Thank you very very much :D

[@Razor_Burn]

Thankyou for sharing another helpful ESPHome integration and I look forward to trying it out once the ATOM Speaker becomes available for sale again. Greetings from Australia!

[@BeardedTinker]

They did sold out very quickly all over the world, but hopefully they'll be in stock soon.

[@tarunarya1780]

Thanks for this video and great tutorial with advice on the pitfalls which is where I got stuck .Lets encrypt then caused issues for use of my HomeAssistant Phone Apps as I had really not wanted to use it with remote access for the users (ideally)

[@BeardedTinker]

I fixed my issues by using ngnix for SSL and it's used both internally and externally. Although under the hood is just plain http...

[@tarunarya1780]

I wanted to just confirm things. I am not sure what you mean regarding the fact that under the hood it is just http. I assume that https causes encryption using the certificates. I installed nginx for SSL. The issue with the phone app is it will not accept a local address and needs the duckdomain dns. This then forces you to open the firewall on the router and then have non-local access. I had hoped to have home assistant not open to the internet via the opened port. I did not try it with the PC as I needed it to work with the home assistant app on my mum's phone. This did not allow an ip address to be used and I was not aware how to alter the "hosts" file on the phone. I would have ideally added a dns server to the pi if it was just a pi, but docker is not something I understood. The issue with the PC home assistant web page and trying to use voice is as you said, it needs an ssl to work. This will probably work locally just using the ip address in the browser, or altering the windows host file. Any help/pointers would be appreciated if time. @@BeardedTinker

[@drwhowhogrub3908]

Thanks for the video, I have it working in docker!!

[@BeardedTinker]

Awesome! I presume that after a bit of pain to install, it works as expected?

[@catcam]

Great video again ! Finnaly migrated my home server from HP workstation aka "flamethrower" to two more eco friendly one liter class pcs. Thanks for inspiring content.

[@BeardedTinker]

Nice! I finally started to measure my desk and looks like it's using 1/4-1/3 of total power, so should look how to lower that usage. Hvala Nikša!!

[@catcam]

@@BeardedTinker 200 w for 5+ years 💀 now is about 60 w

[@BeardedTinker]

@@catcam I've just added power monitoring in my "rack" - well, yeah. Was turning lights off - but yeah - 180-190W is what my NAS(s) and networking equipment use.

[@chrisdixon5241]

Another thought just struck me. From the part of your video on VoIP, the adaptor device looks to just be turning a regular phone into a SIP solution... meaning any SIP phone could potentially be used. Also (the actual thought I had), HA already has an Asterisk integration, so potentially it is possible to set up HA so that you may use the Assist feature from any phone anywhere in the world via Asterisk, and control your house remotely via voice (with suitable security considerations of course!)

[@BeardedTinker]

Yes, but I also think there was a specific request on encoding here that needs to be met - not sure what was it. But, yes, you should be able to do that I think.

[@chrisdixon5241]

Concerning the triggering, I wonder whether an event is sent to HA on button press to say start listening, or whether the button press just tells the ESP device to open the mic and stream the audio to a special HA endpoint. - If HA receives events - listen start(media device speaker0), then listen end. Maybe there is a way to trigger this from something else - If it relies on streaming, then maybe using a sensor to run a HA automation to trigger an ESPHome device to start a new SIP call... though this would involve latency of making a new call

[@dannythedabbler]

I'm looking forward to the day when i can call the HA voice assistant from my car and issue instructions. Really blue sky would be for it to read/write text files so i could integrate with my Obsidian note vault but i think that might be a ways away.

[@typxxilps]

OffTopic: I tried to find a guide about the what kind of home Assistant to choose in 2023 ? I know it could be a question about hardware Pi 4, old W10 Laptop, or what not and native installation or docker and then what kind or even Hassio . I had started in 2020 or 2021 with a raspi installation using a docker image and used it mainly to monitor energy consumption and partly to add some sensors and monitor some data before I had collected enough energy consumption data when the system broke due to missing memory , too small 8GB SD Card. Since then I had no HA running and now I wonder what to choose. I have Raspberry Pi 3 and Pi 4 with 1 GB, old Win 10 Laptop with 4 GB and I will get the basic ihost soon and was hoping to run HA in docker on the ihost for more or less energy monitoring, collecting data like wallbox and Solar power system, battery and maybe weater and Split AC mainly for heating during winter. Yes, I have access via python to I guess all devices cause I used that for a basic quick and dirty powershell dashboard showing the solar power forecast , data of the weatherstation and solar power production. For most of them if not all a HA integration seems to be available therefore HA, but what kind of installation cause I can remember those days back then when I had done my first installation that some had told me why I had not choosen hassio, which should be far better and what not. Therefore the question which way to go or where I can find the ultimate guide to prepare my decision while I am working on the new upgraded solar power system. thanks a lot

[@BeardedTinker]

This is awesome question that's very hard to answer. For example - I would probably go with Laptop and install HA OS on it. RPi 3 is very limiting and 1GB RPi 4 would probably also have a bottleneck due to RAM: Using OS would be my 1st choice as it makes things much simpler to add than Docker. But iHost with Docker is also a good option from what I've tested. Although limited in terms of file access and ease of use. While HA OS or Docker are similar in terms of core HA functionality, there are some things that Docker can't do - for example RESTORE doesn't exist in Docker. Or you can't simply install File Editor/VSCode to edit configuration files. Sure, you can install in Docker, but takes longer, doesn't have all integration features etc.

[@VaibhavShewale]

ooh, it looks cool. dont even knew about it!

[@BeardedTinker]

It has been improved last month and the next release will also bring some new changes.

[@jedivino]

thanks for new video looking forward to trying

[@BeardedTinker]

Thank you 😃 and have fun testing it.

[@POPOL2999]

Hello, it would be nice to have a tutorial with a "standard" ESP32 and inmp441 for example, because cant find anywhere Atom Echo...i try by myself with esphome code but seems no speech recognize. Great video with all info needed, like all your videos thanks!

[@BeardedTinker]

Thanks for the comment 😃 From what I've read on GitHub, support for some devices is still getting added to ESPHome. Not sure if INMP441 is supported out of box. Check here for example: github.com/esphome/esphome/pull/4775

[@POPOL2999]

@@BeardedTinker thanks for your reply! yes i already follow this PR lol, i have to wait, because the PR have Core changes and it has to be merged in EspHome AND in HA to work. But a user has test it with docker image and it work with INMP441, who is really cheap. i really like to watch your videos, very insperating ones! ans clear, with all info and settings needed. Continue your amazing job! lol

[@POPOL2999]

finally it works! i have just forget the adc component in the external_component config! lol

[@BeardedTinker]

@@POPOL2999 that's awesome!

[@miguelavello7493]

Great video as usual. Awesome job explaining and using examples. Anyway that you can use your knowledge and share the seup with ESP32 | INMP441 since the ATOM is unavailable ? Just an idea

[@BeardedTinker]

Thank you!!! From what I've seen in ESPHome documentation, that board still is not 100% supported. I think there is some test support but I don't have the module to test it currently.

[@cimburlen]

@@BeardedTinker Thanks for the great lessons. Today I built a voice device based on ESP32 and INMP44 with PTT button. In the ESPHome integration, everything plays and looks like you have with ATOM, except for the player, but nothing responds to voice.

[@miguelavello7493]

@@cimburlen Thank you so much for sharing. Please keep us in the loop. Awesome that you are already tearing and trying. 👍👍👍

[@WINGNUT307]

Great video. I now have Assist working and can issue voice commands to it from my desktop windows PC (where, i'm guessing, it's using the microphone from my webcam), my laptop (inbuilt microphone) and my phone running the companion app. I also have a much better understanding now of what is going on. I wonder if it's possible to have a button on a dashboard that gets it to listen? Also can it listen via a microphone plugged directly into a NUC running HA?

[@BeardedTinker]

This is something that was promised for future releases. Not available at the moment.

[@e2nomy]

Can't wait friday for the official 2023.5.0 release :).... Are u sure the accelerator of your yellow box is used coz on RP4 it's also about 6s to process a voice sentence..

[@BeardedTinker]

Yellow is running with CM4 module - and that's actually RP4 - so it should be almost the same. I can't say I know of any accelerator for voice in Yellow. it has internal NVME and Zigbee support, but nothing else from what I remember...

[@chrisdixon5241]

Excellent tutorial video, thank you! I'm really impressed with the progress that the HA team have made on bringing us voice control. Personally I will probably go for the ESP solution with several placed around the house... just need the wakeword feature to make it fully usable but I could live with pressing a button in the meantime. I'll probably also have 1 or 2 tablets centrally somewhere on a wall as a dashboard running fully kiosk browser, again wakeword would make these more usable. I wonder if there could also be a way to make them listen when the screen is on or someone is close by (camera or proximity sensor on the tablet)?

[@BeardedTinker]

Thank you!!! Did you try with mobile app? HA Companion App - and Assist button also works. I'm not sure if wakeup word would bi similar to Rhasspy or not. There I used Atom Echo with wakeup word and it worked really well... But this has been implemented a bit different as it doesn't send voice back to HA until you release the button. That part needs to change. As for tablets, you could potentially do that. NSPanel Pro does register back proximity. We just have to have a way to trigger mic button as now it requires fingers :)

[@chrisdixon5241]

@@BeardedTinker I did try out the HA mobile app before the year of the voice started, already the text to speech was working rather nicely to trigger some things in HA, but I didn't have time to play more recently. Unfortunately I don't have any ESP device or smart speakers... on my list for when I have some free time... family life keeps you busy, I'm sure you know! :) but I have been watching Rhasspy with interest for over a year and imagine that HA will take a similar direction, especially since they hired the developer :)

[@vimvigour3327]

Can you teach us how to get a valid SSL certificate? Is the process the same if we just want to access HA locally?

[@BeardedTinker]

Setting up SSL certificate depends on you r install type but also some other things. The easiest way to get is to use Let's encrypt and DuckDNS AddOn in Home Assistant and by enabling NAT loop back on router domain name to access it. Other option is to us NGNIX plus Let's encrypt - I use that myself but my setup is on Synology and NGNIX is available through UI there. But it's really dependant on how your overall setup looks.

[@vimvigour3327]

@@BeardedTinker Thanks. Sounds like we need a tutorial to cover those options.

[@snappytom88]

@@BeardedTinker excellent video! would you know if the use of the Atom Echo (or any ESPHome device) using whisper and piper needs to use https? or is https just required for Home Assistant Cloud?

[@BeardedTinker]

@@snappytom88 you don't need SSL certificate for ESPHome (Atom Echo). Currently it is only required by browser if you want to use microphone on your PC for Assist.

[@bojang.8423]

Jel se može više atom echo uređaja postaviti na home asisstant? Tako da imam na različitim područjima doma?

[@BeardedTinker]

Da. Može ih biti više, ali zavisi od sustava na kojem se vrti. Prema info koju imam, RP4 sa 4GB RAM bi trebao podržavati do 5 Atom Echo. Nisam probao, pa ovo samo prenosim informaciju od developera.

[@bojang.8423]

Hvala! Naručio sam ih dva, pa ću vidjeti. Hvala na videu, potaknuo me da kupim atom echo i sve posložim. Sve će raditi na i3 brix, pa valjda bude ok.

[@BeardedTinker]

Trebalo bi sve biti OK - ovo sa 5 komada to je ako se koristi i Wake-up word@@bojang.8423

[@Hafenstrand5]

Is there by now anything else available beside toggling lights? I wonder if I can set timers or ask for a value (current energy consumption). I didn't had any success so far.

[@BeardedTinker]

Yes, for example turn on/off switches, ask for temperature etc. Plus you can always create custom intents for anything that's not already available.

[@yousaf.saleem]

microphone doesn't without Https link to homeassistant, which is almost impossible with one being behind CGNAT.

[@BeardedTinker]

Can you use ngnix to convert traffic to HTTPS? But yes, as I've mentioned in video few times, valid secure connection is required. And that's not limitation of HA, but browsers that don't allow mic access from uncensured sites, to protect your privacy.

[@yousaf.saleem]

@@BeardedTinker so esphome based mics work without that issue? bcz I can build a INMP441 I2S based esphome mic, infact I was the first one to point it to jesse.

[@BeardedTinker]

You can try. AFAIK ESPHome still doesn't fully support INMP441 - there is issue open, but I think nothing has been merged in released.

[@jameshendry3571]

It would be great to be able to call the assistant from lovelace rather than going to the three dots each time. How would you do that?

[@BeardedTinker]

We talked about it with devs. AFAIK it's not supported right now, but will be probably added very soon.

[@jameshendry3571]

@@BeardedTinker it would be pretty cool to go to a tablet, hit a custom button card or the like and have a conversation with HA. Love the content BTW

[@BeardedTinker]

Yes, that would really help a lot. :) Thank you :)

[@chrisrosenkreuz23]

wait so the STT can be set to use any connected mic and uses the physically connected one on your HA server by default? and can be triggered by a connected button? if so, there are ways of making a physical button on your mobile phone act as a regular old "smart button" type and have it trigger the STT then use the mic on the mobile to issue commands

[@BeardedTinker]

From what I've tested - on mobile phone and mobile app (Companion App) - you can start Assist, press on the mic, but in never registers the voice. App does have permission to access the mic, but nothing gets picked up.

[@BeardedTinker]

Hmm - actually, after bit of tinkering, it works. Not sure what I've done, but now it does recognise STT and can control devices. So, answer is, year, you could probably do that 🤣

[@chrisrosenkreuz23]

@@BeardedTinker wow that's amazing, THANKS!

[@BrianHCohen]

Now we just need a way to do that from a smartwatch (like the wear OS based GW5).

[@chrisrosenkreuz23]

@@BrianHCohen does the companion app not work on it? dunno, I don't own one. If not, then you could try create an actionable button blocks via the google API, but I have no idea if the buttons can be binded to the HA action of pressing that STT microphone as BeardedTinker described above. I know for a fact they can be binded to regular HA actions though.

[@darknessblades]

Is there a way to use the M5stack echo with something like a ZIGBEE button to remotely trigger the button? Since its just a trigger button, it should be possible to remotely trigger it

[@BeardedTinker]

Not yet - its not trigger button (still) - you need to press and hold it while talking. But next version of ESPHome will have what you need.

[@darknessblades]

@@BeardedTinker Looks like I will test it out with the current version first {got a M5 echo for like 15 euro with vat}. so for testing it out I think the first one would be fine. ---- I am also gonna make a mini statue just for fun [Was thinking MO from WALL-E]

[@BeardedTinker]

In current release you need to have it triggered whole time during the speech.

[@Badg0r]

I have a dell optiplex 9020 with an i5 processor, 32gb DDR3 and a SSD running home assistant. Will the local voice conversion be faster on a system like this? I've just bought the grand stream 801 for fun.

[@BeardedTinker]

It should - can't tell you exactly how much, but fast(er) disks with more RAM should definitely improve performance.

[@Badg0r]

@@BeardedTinker thank you for your reply. I'll let you know what the outcome is after I've found a cool and unique looking phone :). The phone has to be a real retro and funny one.

[@OnlineObsessionista]

The Nabu Casa assist doe snot work out of the box as of Aug 1, 2024. And there is no way to get support on this issue.

[@BeardedTinker]

I missed this one - what exactly is the problem?

[@OnlineObsessionista]

@@BeardedTinker I'm getting: sorry couldn't understand that on anything I'm saying. I'm using all default setup with nabu casa

[@BeardedTinker]

@@OnlineObsessionista Does same thing happen if you speak in mic and also type? I've tried it her eon my side and works OK.

[@OnlineObsessionista]

@@BeardedTinker it happens no matter I speak or type. But I think I know what the issue is: the assistant is so dumb it only understand a particular command and you need to expose the sensor beforeahead. It cannot answer things like: hi, what can you do, etc.

[@BeardedTinker]

@@OnlineObsessionista it doesn't need to be that way, but does require you to configure OpenAI or if you have very powerful machine available Local LLM. Unless you have AI system in place, yes, it will only answer predefined sentences or if you create your own

[@ThePakovia]

can i use Sonos speaker for this?

[@antoniodecrisci2369]

Hi Do you know if Super Mario is the only character that can be used for the OpenAI prompt? thanks

[@BeardedTinker]

From what I remember with call with devs, you can setup your own character. But I missed a lot of the call as I was walking in busy street while on the call :) They did say how it was setup, that part I missed.

[@antoniodecrisci2369]

@@BeardedTinker Ok, no problem. I think I figured it out! You have to place it in the OpenAI prompt text box (in configure), for example, I placed "You are Anna from Frozen. Be funny." for my daughter. Now, select the OpenAI voice and it now answers as Anna from Frozen. The voice itself did not change to Anna BUT it knows everything about Anna. If you wanted to keep other characters, I had to re-add the integration and that made a separate conversation agent using another API key. That way you can select any OpenAI voice agent you want in the drop-down menu (pipeline), otherwise, you just have to keep changing the voice prompt.

[@BeardedTinker]

Voice itself is an issue of course due to copyright for those characters. Glad you got it sorted out!!!

[@Goldman5800]

We need a solution to use microfon without pressing any buttons ore voice activation command like hey Google

[@BeardedTinker]

Wake word will be added later in the year according to the devs.

[@Goldman5800]

@@BeardedTinker that's great

[@JoshFisher567]

Rhasspy works with a wake word although setup is a bit more 8nvikved. Works with a USB speakerphone plugged into you HA and they make hats for raspberry PI's for satellites. Atom Echo also works with Rhasspy. With that said, Nabu Casa hired the guy who wrote it so don't expect any updates but it actually works very well and has for months but at this point I would wait for Nabu Casa solution. Had to get rid of my Alexa after finding out about Amazon Sidewalk. Super sketchy but Amazon is only doing it in the US (currently). They have built a BT and LORA mesh network that covers 90 percent of the US population (people, not the size of the country. You can disable it but you have to for every Amazon device so if you have a Ring and Echo you have to disable it in both. Uses triple encryption and Amazon "swears" it's only being used for controlling IOT devices but I don't trust billion dollar companies. If anything it should have been opt-in, not opt out. LORA radios can reach half a mile and use 900mhz frequency and are in newer echos , ring doorbells, and fire sticks (or any other iot device Amazon makes spore owns) and outside sidewalk there is zero reason for them to have it so Amazon has built a mesh infrastructure of devices people buy with zero investment in infrastructure. Sure, nothing but for the good of it's users even though almost nobody knows about it. Essentially, an example would be if you have a Wi-Fi light that has a better connection to your neighbor's router. It would use both your and your neighbors internet then your neighbors Network would turn the light on instead of your own. Nothing shady about that. Especially considering it's using both yours and your neighbors Internet without any permission unless you opt out in every app for every device you own controlled by Amazon and now they are letting 3rd parties get access.

[@smartifi_egypt]

Thanks so much for the video and I really like your explaining approach. Just a question, after installing the add-ons whisper and piper, they exposed an entity that always has a state unknown, is that normal?

[@BeardedTinker]

Thank you for the comment - really appreciated it! In terms of entity - yes, I also see them as integration with 1 entity. Piper is not empty and whisper is unknown. That's I think normal as they both work OK for me.

[@smartifi_egypt]

@@BeardedTinker good since I have a delay >20s and I run HA in a virtual machine on my powerful i7 laptop I am trying to figure out why

[@BeardedTinker]

I also run in VM and on much lower end machine but have around 4-7 seconds delay. Check vCPU and RAM you gave to VM. I gave to vcores and 4GB RAM.

[@Jcarlier50]

can't get other languages in a new assist ( ihave made AUTO in whisper) !!!!!!!!!!!!!!!!

[@BeardedTinker]

How come? Try creating new Assist in voice assistant menu, and select new language. I have 4 languages currently and they all work.

[@kloakovalimonada]

I'm sorry, I know this is a hard problem to solve, but this is like speech recognition in the late 1990s - totaly unusable in practice. I love HA and been using it for 5 years but this is laughably bad.

[@BeardedTinker]

There were a lot of updates after this one, including custom wake word, your own voice for STT etc..

[@barkerd427]

Why is Czech so fast compared to English? Is it your accent?

[@BeardedTinker]

Depends - did you use Nabu casa or local, what machine are you running it on? For me Nabu casa is instant, and if testing on Yellow with 2GB RAM, with local it is very slow.

[@Tntdruid]

I dont use any voice stuff in HA 😂

[@BeardedTinker]

Noooo :) But you'll return to the path - this is the way...

[@Aurleis]

@@BeardedTinker "This is the way" My only wish is that we could do memes for replies #Mandalorian #ThisIsTheWay

[@frankraym]

Since installing this it made my home assistant super slow and missing automations.....4/10 for me

[@BeardedTinker]

Missing automation? Never seen such a problem? Did you look at Github if anyone has reported this and if not - try to open ticket yourself?

[@ToTo-od9wz]

I think it is stupid to use another device when we carry all the time a phone with us. Also, we have Google Nest in every room...

[@BeardedTinker]

To each it's own... What stops you from using phone...

[@nebulousJames12345]

strange that this would be released without wakeword

[@BeardedTinker]

Why do you think so?

[@timtaler8439]

Sorry, but the year of the voice seems to be a wasted year. Too complicated, too slow. Maybe next year.

[@BeardedTinker]

Too slow? From my experience, it answers in less than a second and command has already been triggered.

[@smartifi_egypt]

Sorry but strongly disagree 😊, controlling your home by voice is a game changer in the smart home industry, I don't need to turn the light on/off from my phone instead of the already available and easy to reach wall switch but using voice I am convinced 🙏. One year is not so much especially when we know that giant techs Apple, Google and amazon have been working for years with huge resources

[@dkickelbick]

Is it possible to setup a Home Assistant system in such a way that I connect securely to it by a local address (ip or homeassistant.local)? Can I get a valid local certificate? I mean, I have Nabu Casa cloud and connect to my system using it, but I like to use independent of my Internet connection. If there is a way, I would like it very much if you could make a video about it. Apart from that, thanks for the good work!

[@BeardedTinker]

Setting up valid local ip certificate could be possible but it can be too much pain as it also has to resolve to FQDN. I use ngnix and NAT loopback so I can resolve valid domain name internally. But this again depends on the network you have. That way my domain is resolved to local IP address and I can use secure connection internally too. Previously I was playing with local SSL but it was just too much pain to setup and maintain.

[@dkickelbick]

@@BeardedTinker Thanks a lot. It sounds not useful to do for me. Too much hassle. Thus, I will go via cloud or use a hardware button.