The cost to use OpenAI in your Smart Home

[@KeithConley]

Thanks! This seems to be a direct answer to my question on your last video.

[@technithusiast]

Hey, no problem! Glad I could help.

[@poshsims4016]

Maybe using langchain will help the memory state & controlling it better

[@johnlambert7405]

Also adds some nice abstractions to use locally hosted LLMs.

[@technithusiast]

I never worked with langchain but it looks interesting. I’m working on creating a way to use local llm in automations but I’m currently having trouble getting the local llm to work😅

[@MacDork]

ooh ty. been wondering this too

[@technithusiast]

No problem!

[@PontusWelin]

This gave me an idea! What about doing two calls to the API? One using register intent/call intent nodes to figure out what it want to do. But this uses as little data as possible. Then when you know the specifics you can make another call where you add the relevant data. You’d get the best of both worlds. Except it will take more time potentially. Though I’m not even sure it would since the big api calls take longer anyway.

[@NigelinAustralia]

I had the same idea. Get gpt to work out the intent of something. If there was a way gpt could remember context of your house and what devices were there it could work out intent without having to have updated states initially

[@PontusWelin]

@@NigelinAustralia i feel like that’s what the assistant api is for basically.

[@technithusiast]

I like your thought process! I liked the way this conversation in this thread was going that i had to create a quick video about my thoughts which you can see here: kzbin.info/www/bejne/d2TYiaqQacScgJI TL;DR - I had a similar idea of separating the calls last fall and rewrote Ke to use this technique and.... it wasn't as good as i thought it would be.

[@videoblight]

It’s a good idea to set up rate limits. I had local voice with extended open-ai on a spare Pi and it went haywire. It sent thousands of requests and chalked up $20 in less than a day.

[@technithusiast]

Yup not surprised. The extended api is really impressive but the trade off is that it’s greedy

[@PontusWelin]

I was also wondering if it is possible to setup an assistant with the assistants API to not have to send all the data all the time? Like you wouldn’t need the system node and the template data could be greatly reduced probably.

[@NigelinAustralia]

I asked similar. The challenge is the assistant api requires far more conversation management at the moment. Works brilliantly in sandbox but actually calling the api kind of sucks

[@PontusWelin]

@@NigelinAustralia ok. Conversation management how?

[@NigelinAustralia]

It's overly complicated at the moment - hoping they simplify

[@technithusiast]

Correct. I think the Assistant API has promise but as it stands, its complicated and nature of Smart Homes Require that the AI is aware of the state at all times. This mean with each call you have to let the AI know the state of everything. In the end you will always have to send a ton of data, unless you know before hand that what you are asking for does not require a lot of data.

[@technithusiast]

I just posted a video in Chat that talked about this a little more: kzbin.info/www/bejne/d2TYiaqQacScgJI

[@aidanb8719]

I have been looking at the litellm and using the proxy you could have it call that and then it might be able to get the response back from a range of LLM some local some in the cloud. Just the local might take some time but if all its doing is returning information you pass in then I can't see it being too slow. Worth a try and if works then it's all free.

[@technithusiast]

Can you elaborate a little further what you mean? i am not sure i quite follow.

[@aidanb8719]

@@technithusiast you can host local LLM and there is something called litellm and that will allow you to make api calls to the local LLM or to cloud-stored APIs its useful if you want to use some coding extension such as GPT-Pilot you point the config to call the proxy then it will call where it needs to. A lot of people on here explain it better than me but that's my thinking that you could use that as it would not need the cloud and its all local so safer too.

[@dragoon347]

Couldnt you just hook this up to an internal house server running a small phi3 model?

[@tedev]

unless chatGPT is able to store data and status of things, it will be really expensive and time consuming in the long run. i don't want to send an entire list of all my lights everytime i want to say "turn the lights off in the living room". i want to update it once and keep it. maybe a session would work. for example if i start a chat with GPT.... i can list all my lights and then keep talking to that chat later on...

[@technithusiast]

Technically, you can accomplish something like this but the experience degrades over time. As a simple example, Imagine if you had two lights in you house that were off and you send GPT both lights and their states. Then someone manually turned on a light. How would GPT know with out you sending the state of everything. Now imagine we ask GPT to turn on a light and it sends the command to you but HA failed to turn it on. How would GPT know? I totally agree that it sucks sending all the info every time you call GPT but the way the LLMs work gives us no choice. AI-intent does have a small advantage here where you can send GPT specifically what it needs based on your commands but it requires clever planning.

[@OldManShoutsAtClouds]

​@technithusiast the API call to chat gpt should be pretty much the last in your chain. Prior to that, you should build the context using other more standards API calls. Just ask HA for the status of all devices and include that output as context. The biggest thing you're missing is context. "It's dark".... where? It's dark in the living room... not your daughters room, not the kitchen the freaking living room. Use. More. Context.

[@BGraves]

Might be naive to think these prices aren't introductory rates...

[@technithusiast]

I’m certain these rates will go up over time too