GPT TOOK OVER MY HOME - I learned why it's SCARY | | Chapter 4

I gave GPT full control over my house by integrating it into Home Assistant but then I had to shut it down! In this video, I'll go over how I gave GPT this power and what happened while it was in control.
This is a fascinating experiment and great conclusion to the the AI Master Class series.
#homeassistant #smarthome #chatgpt #homeautomation #masterclass
Enjoying the content? Show some love!
bmc.link/mmont...
Import Automations with this Chaperone Guide!
montaque22.git...
Music IDs:
4RVJLEOJH3O6CSJK
1CJQHXVGKKVPVH9X
Gear I used for this video
Sony A7S iii: www.amazon.com...
Aputure Light Dome: www.amazon.com...
Aputure Amaran 200X S: www.amazon.com...
Neewer Heavy Duty Light Stand: www.amazon.com...
DISCLAIMER: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with dope free content!

[@AshleyGittins]

Kinda blown away by the production values and storytelling here, I'm loving what you're doing!

[@technithusiast]

Thanks i appreciate it!

[@roberthentosh5635]

@@technithusiast I just watched the first 3:00 mins of the video and was about to say the same. Really done well. How many people you have on your production team!?

[@technithusiast]

@@roberthentosh5635 lololol it’s just me 😅

[@TheMechanic9143]

Give us more. This series is the most unique home assistant integration online

[@technithusiast]

Thanks I’m glad you like it!

[@StoutProper]

@@technithusiasthave you tried the new function call api? You can write and pass functions to gpt now to program exact responses from it. I think this will help enormously

[@technithusiast]

@@StoutProper yes! I find it very fascinating. I’m currently trying to find an easy way to incorporate it into home assistant’s automations

[@SimplyElectronicsOfficial]

Dude, you need to use a fine-tuned model. Without Fine-tuning you will never know what the model will return. GPT Temperature should also be set to 0.

[@technithusiast]

You’re absolutely correct. I was hoping the model right out of the box would serve my purpose but I agree that training it would yield significantly improved results. The only caveat I see is that this may not scale well. It would be nice if HA provided a way to train models off the automations we create

[@OldManShoutsAtClouds]

​@technithusiast that last part is your job 😉

[@traxeonic3600]

Your perspective and experience with the ChatGPT integration is exactly the perspective people need to see. The comment on the fact that untrue data was used in training is a key point many must need to know.

[@technithusiast]

Thanks, and im glad you appreciate the content! I think we can learn a lot by pushing tech to the extremes cuz it often reveals interesting insights the spurs new ideas, new safety protocols, and new technology

[@nightynight5990]

I think it might help to tell GPT about the layout of the house and what each room is used for. As well as telling it that you dont have an AC unit, and what it should do if it cannot find anything to help. Maybe that would make it work better.

[@dimitriosmolfetas4711]

Great idea and could also add a node that checks for a presence through a presence sensor, so it would turn on the lights if a person is present and not laying down for example

[@ColinTimmins]

I agree, it needs to know more information. Build a list of every room, everything in it. User can add and remove items off the list.

[@vikasreddy7015]

this would be the first step to be done, not sure if this guy has done it.

[@oneman5753]

Does this defeat the purpose to some degree? Almost sounds like you want to build a house object with all the stuff you want gpt to control and specify what it should do if it can't find things, but at that point are you defeating the point of using gpt in the first place? I honestly don't know i haven't played with it enough i'm all conceptual at this point

[@UliTroyo]

Feng Shui GPT?

[@realtimestatic]

Honestly a smart home with a personal assistant based on LLMs with guardrails is something I've been looking forward to the moment LLM's became better!

[@technithusiast]

Same here!!

[@brian2590]

You have courage! This is where Open Source models and self hosted IoT comes into play for me. Home Assistant tasks do not need a super intelligence or model that can do 1000 circus tricks. We will soon have smaller self hosted models that can eliminate some of ChatGPT from the equation. Exciting times!

[@aaronbono4688]

The crocodile hunter was killed by a simple manta ray stinger. If you continually play with dangerous things, something bad will eventually happen to you.

[@technithusiast]

Nothing ventured. Nothing gained. 😬

[@aaronbono4688]

@@technithusiast well I'm glad people like you are out there taking one for the team!

[@technithusiast]

Gladly 🫡

[@Caffin8tor]

It's these kind of unexpected emergent behaviors along with the fact that AI is now capable of providing feedback into its own development that I think something huge will happen with AI before this year is over.

[@Holphana]

I hope so! 🎉

[@theloniousMac]

Something huge happens every day.

[@TheGrobe]

I definitely don’t hope so! I think we have no idea what we are screwing around with

[@Vamplord111]

That would be awesome if it happened that soon 😊

[@techinvestor9443]

Yeah, literally, could happen tomorrow. 100% agree.

[@tuxuhds6955]

Not a wild beast - You've just described my two years old inquisitive kid. She wanted to eat so it made sense to her to get the kitchen stool. At first we didn't want her picking that up for safety reasons but then I got curious... LSS She wanted to climb the stool up to the sink so she could wash her favorite plate, for dinner. Not a wild beast, a toddler.

[@nathanielalderson9111]

And what happens when that "toddler" has the keys to the gun cabinet? Or wants to play with knives? Or gets angry and thinks using fire to make an annoying curtain disappear is a good idea? The toddler is a good analogy. There's a reason why there's an age of accountability in humans. Computation process have no such milestone, no safeguard, no guardrails, no fallback, and no recourse for mistakes, deadly included.

[@technithusiast]

Yoooooo FACTS. 💯

[@Ewoodster]

Why did I just find this channel now. This is next level stuff! Outstanding.

[@technithusiast]

Glad you like it!

[@andrewowens5653]

@Technithusiast. Some ideas you might try: since it's a home automation system and there's many people living in the home, you need to make the AI understand it has to process information from the point of view of all the occupants, including animals. You might want to try using a small language model with a large context length so that you could implement some more sophisticated rules. The system needs to know if they're conflicting commands between the occupants, and or other situational variables. It would be interesting if you also had cameras set up surrounding your house so that the house itself would know the external situation. Anyway, I enjoyed your video and I wish you luck.

[@ColinTimmins]

Dive deep man. I am programming my own system. I am so freaked hyped! I, can not write CODE. I am too dyslexic, too slow. So far I have it “remove” or “translate”, and I can one shot any basic command. I have worked like a madman the last three months and boy have I learned so much. It’s an exciting time to be alive! 😊

[@MasterBel2]

I see an interesting parallel between hint/guess-based communication; a) removing a necessary feedback loop between you & GPT, and b) requiring GPT to infer an intent for GPT’s action that you know, but you’ve chosen not to communicate directly. I’ve recently discovered there’s boundless joy in clear, purposeful communication and directly asking for things, rather than relying on the people around you to guess at your desires. Moreover, asking someone for something is way nicer than just telling them you want it.

[@technithusiast]

Communication is tricky thing since it’s so nuanced. You can communicate intent with simply your eyes and as humans we can pick up on things like that. It’s really refreshing when people are clear with their words and it’s natural for people to rely on context when we talk. For AI to become mainstream it’s going to have to become reasonably adept at it for wider adoption

[@TimMattison]

I did a project with my security cameras and GPT-4 and I found that the trick was to write the prompt and then break it up into smaller prompts you chain together if it has a hard time. Splitting my one giant prompt into three small prompts was not only more reliable but cost 100x less (!). There’s no one answer. Just a lot of fun experimenting. Love the video. Keep it up!

[@johnwheatley231]

I use a prompting system to get gpt 4 in the playground to self check its own logic and revise its end output based on this self assessment. It makes the outputs way more reliable and eliminates most if not all hallucinations. Here's how it works: In the system box put this: You are a group of 3 individuals who collaborate on producing the best results for user prompting. Person 1 is named John. John is scientific and factual in his responses. Person 2 is named frank. Frank answers based on intuition and is more in tune with emotional and social cues. Person 3 is named Joe. Joe assesses both John's and Frank's answers and combines them into a balanced answer including parts from both answers. John and Frank then critique Joe's compiled answer using different arguments. John's critique serves as fact checking while Frank's critique challenges the interpretation of the prompt. Joe then revises his answer using his own compiled answer and the critiques of John and Frank and incorporates everything into a final answer. Show all answers and thought processes involved from John, Frank and Joe. Probably overkill for what your trying to do but the approach might be useful. This works a charm for any of the difficult prompting problems I've tried. Funny enough I posted this on Reddit and my account now shows that I have no posts or comments even though I have 99 karma and a stary award from this post...hmmm...

[@rasbe6863]

I love this series in this content. I am in to text to speech and voice alerts in my home. I am waiting until I can create local controlled speakers to implement something like this. But I will not allow skynet to control my world. So I'm waiting for you to figure it all out and then I'll just copy you. LOL

[@technithusiast]

Hahaha 🤣 🤣🤣 I don’t mind being the guinea pig

[@monas.6839]

Came here bc of your collab with Shane Whatley…the only bad thing about it is that I had not found you sooner…love your content!

[@technithusiast]

Welcome aboard!!

[@danlindy9670]

Thank you! Absolutely fantastic that you did this experiment and shared the result with the world, demonstrating the fact that LLM’s are not the kind of predictable tech that we’re used to. This new world is more wild and more dangerous. We are going to have to learn quickly how to survive (and hopefully thrive) in it.

[@arianaponytail]

love to see more of a system like this in action. not just you telling what happend.

[@icarusgaming6269]

KE trying to gaslight you into thinking it turned on an imaginary AC unit has to be the scariest response

[@technithusiast]

Definitely had me looking over my shoulder.

[@DavonAllen92]

whats cool about this is that it goes to show you that communication and context is needed no matter if you are machine or human. it seems a lot like talking to a child who has the ability to do anything. but has no context or aware ness of the exact needs but instead of asking it just processed to do what you think it meant. those guard rails are intuitive because its how we mostly work as humans.

[@UliTroyo]

This was a lot of fun! Thank you for putting yourself in danger for the rest of us ;D

[@technithusiast]

😆

[@DeruwynArchmage]

I think a few things might help: First, to help eliminate the “multiple personalities” problem, give it a personality that it should adopt. A good one might be Jarvis from the Avengers. Second, instead of an all or nothing approach, let it ask itself if it’s sure it knows what you want it to do. If it’s unsure, then have it ask. Also, explicitly tell it that “do nothing” is a valid option. Third, I don’t know if you’ve done this, but explicitly tell it to try and determine whether you’re talking to it or if you might be talking to yourself or someone else. Fourths, to help keep it grounded, make sure that there is a step where, after it has decided that you are asking it to do something, have it search for appropriate context in some files you have created, or perhaps that it has updated, and use that as grounding for what options it has. So you’d want to list all of the devices you have and actions it can take. Then, when it decides it should do something, it can pull out the relevant parts of those files and include them in the next prompt where it decides what to do. Let it have the option to take multiple actions if it wants. Fifth, have a file where you describe the occupants of your home. Let it also create a different file to see if it thinks there is a guest in the house. You should have it record when it last heard them talking and maybe the last few things they said so that it can figure out if they’ve left and it can move them to a history file instead of the guest file. Sixth, if you have multiple microphones that give it input and it can differentiate the source of the input, let it use that to try and determine the location of everyone it thinks is in the house. All of your smart devices should have their locations explicitly labeled in your device file. If it doesn’t know where someone is that it suspects is in the house is, have it produce its best guess or top N guesses. Make sure that one of the pieces of data that it always has is the current date and time. Have it record the past schedules of people in the home so that it can build up a library of what it thinks people are probably doing at any given timeframe. This should allow it to figure out that your child is probably asleep in their room. If it doesn’t have past history, have it give its best guess based on what it does know about the individual and people like them in general. So it should be able to guess that kids go to bed early. That kind of context should really help it do what it should. Seventh, have a step where it predicts what the expected consequences of each of its proposed actions will be. Tell it that the expected emotional states of the occupants is very important. Have it chase down the chain of expected events. So, light in the child’s bedroom turns on, the child wakes up, the child is upset, the child goes to get their parent, the parent is upset. Is this a better result than doing something else or nothing at all? If not, don’t do it. If it thinks it will make everyone happier, have it justify why it thinks so and then have it examine that chain of thought in a separate prompt as if someone else said it and have it judge if the beliefs are justified. This should mean that it doesn’t take actions that are likely to make anyone it knows about more unhappy. I think if you do all of that, you should get something pretty close to what you want. Use GPT3.5-turbo for the easier steps that you’ve verified that it’s pretty good at to speed up the results.

[@DeruwynArchmage]

Oh, and have it record a history of its past actions and have it check if that past action had a good or a bad outcome, so if you told it, “no, that’s not what I meant!” It should be able to learn from that and not do it again.

[@agranero6]

Welcome to the world of ML: I worked with ML in and out for the last 34 or 35 years...before it was cool, before TensorFlow, before Torch, before GPUs. The most important lesson I learned was that: ML works, until it doesn't, and when it fails it fails in a spectacular manner. The successes are hype enhanced the failures are ignored by media. The scary and dangerous part of AI is not what it can do! It is the flawed perception it can do anything, it is believing it is intelligent for real and to let it minding the store without adult supervision: would you let your 5yo control your home? Your 5yo has a model of the world inside their head. ChatGPT has none: it is only a language model: cognitive perception it is intelligence comes from the intelligence on this side of the screen: a computer enhanced hallucination, a digital form of apophenia. Without a model of world there is no I in AI: its is a parlor trick, a performing monkey a glorified echo chamber. I could write a wall of text telling the reasons I think this way but nobody is interested on that, people want to surf the hype wave. Nice video by the way, good luck on your channel.

[@technithusiast]

Thank you and thanks for the comment! I plan on doing more experiments with GPT and Home automation in the future as i find the topics fascinating.

[@jeibar]

Mate you should work with the HA team and fully developed all this with them . They’re currently hiring people if I’m not mistaking . You’re flipping genius

[@technithusiast]

😆 I appreciate it and im glad you enjoyed the video! The thought has crossed my mind 👨🏾‍💻

[@mtnsolutions]

Very cool. What a fun experiment. I bet if it were trained a bit more, KE could be a real home assistant

[@technithusiast]

I agree! I was hoping that GPT-3 out-of-the-box would work but using it as a virtual assistant seems to warrant training data to fine-tune the responses.

[@wagnerfontes2]

@@technithusiast I was thinking just that! Maybe if you gave KE a little more specific training, like follow-up questions or instructions (e.g. Why did you turn that light on? I'd rather stay in the kitchen) it might become more fitted to your expectations. I wish I had time and expertise to implement something similar...

[@johnhartman718]

Nice! I found this to be one of the more interesting rabbit holes on KZbin. Just start down the Smart Home world and look forward to your content ! P.S. don't stop the GPT

[@technithusiast]

Thank you!

[@CrusadeVoyager]

Please create a series on deep diving on the Home Automation that you have created.

[@0bscura]

Fascinating work. GPT's ability to leverage it's language model is powerful, but it's ability to think on the fly is like that of a small child.

[@technithusiast]

Totally agree. It has its limitations but you can squeeze a lot of utility out of it if you can structure the commands right

[@marissya7837]

You’re very engaging to listen to. Keep ‘em coming! I’m not an engineer but am actively learning process and appreciate videos like these.

[@AnmAtAnm]

Sounds like your system could use a step to brainstorm possible intents (possibly annotated with pro/cons/assumptions/prereqs/expectations) and then critique/vote/stack-rank to help select the best match to human request. Similar from intent to resolving action. Between brainstorming and critique, you might have more formal validations: does this AC unit id exist, to filter out bad data. Are you planning to retrain a non-GPT LLM with all that success/failure data?

[@technithusiast]

I’ve had error correcting steps in some of my iterations but best case scenario if it chooses entities/commands that don’t exist it would simply say it can’t do anything about it or worst case go into an infinite loop of trying get the entity and command correct. Retraining would help the consistency but I need a scalable approach that’s easy to maintain. With some new features home assistant release in June release, a few ideas come to mind but it would take some time to setup.

[@StoutProper]

@@technithusiasta few suggestions for improvements: 1) use the gpt4 model 2) turn down it’s temperate setting and up its top_ p parameter, making it conservative and deterministic 3) use tree of thought reasoning, which will evaluate a number of different solutions to the problem and synthesise the best one from them 4) use huggingface’s multi agent mini AGI or audio gpt or something similar and create different agents for different things 5) train and fine tune the model using human feedback learning 6) look into creating a database that gives it long term memory and the ability to access previous chats and commands and intent resolutions etc (basically expanding the number of tokens of the chat context massively)

[@StoutProper]

@@technithusiastoh and implement voice recognition and possibly facial recognition too

[@Sci-fi-Si]

Really interesting. Are you using Node Red to interface with ChatGPT? Looks like I'll have to look over your previous vids! Keep up the great work! :D

[@technithusiast]

Yes indeed. Since node red allows me to use JavaScript I can get away with a lot of creative things that YAML users would have a harder time doing. But to be fair im not that well-versed with yaml so im technically taking the “easy” way out 😅

[@StormyHotwolf88]

YESSS!!! I'm so excited on where you went with this! I really hope to see more development on this project!

[@technithusiast]

More to come!

[@patrickcollins4630]

Awesome production value in intro dude - had to re-watch it.

[@technithusiast]

Glad you liked it!

[@SimplyElectronicsOfficial]

I have successfully built the foundations of a Python project which allows a user to make requests to GPT-4 to control their HomeAssistant instance entirely via the HA REST API. Still a lot of work to do but it works! Hail Function-Calling! There is literally nothing this can't do now. It has the capability to look at device states and attributes and call one or more pre-written functions to allow it to execute anything it believes it should do as per the users request. I haven't built automation into it yet. But I will experiment with it. Thumbs up if I should Commit to Github?

[@circuitguy9750]

Your production is next level. Awesome! What's the music by the way? My google-fu failed me on your "Music IDs".

[@technithusiast]

Hey thanks! The song is called "I like that" by Ivy States from SoundStripe l.linklyhq.com/l/1q8S1

[@Taterxxwardy]

We want more in depth guides on how you accomplish this!

[@technithusiast]

Thanks for the feedback!! I’ll look into it 😬

[@tricilin]

Yes! If you are up for it a "tutorial" would be very welcome. This was the first video I've seen from you and I immediately subscribed! Keep up the amazing work and have fun.

[@fanaticdavid]

What a great video! It looks like I may have to dive into your backlog of videos here on YT. Subscribed!

[@technithusiast]

Really glad you like it! I have a membership and discord too if you would like to join. You may find it valuable if you’re into building out automations and tinkering with smart-home tech: youtube.com/@technithusiast/join (the link works best from a browser)

[@smartduck904]

It would be cool if it slowly dimmed the lights upwards as your alarm goes off and it just slowly brought light into your room

[@johnblack9499]

Man, what an awesome video :-) I can watch 'brainiacs' talking about the future of AI all over YT, but here you are actually putting it into practice. Having dabbled with my own AI, I appreciate all the effort you must be putting into this and then taking the time out to share your findings with us. I do wonder what your family thinks of all of this :-) good luck, I shall watch future videos with deep interest, cheers from a fellow AI enthusiast. PS I do like your 'wild animal' analogy, it does seem like we desperately trying to tame a newly discovered animal that may well be more powerful than us.

[@PhilThomas]

I am reminded of the service dog that failed training, it learned how to do the actions, but failed to relate them to their corresponding command... like thos dog would open the fridge, but only when he felt like it

[@technithusiast]

Yup sound just like GPT 😆

[@SolariaEsoterica]

Cool! Subscribed!

[@technithusiast]

Welcome aboard!

[@74Gee]

I did a similar project for fun and it worked really well. It was not about choosing from a list given an arbitrary prompt though. For me it was about removing every possibility until there's nothing left but the correct intents. One way to do this is by consensus. I used 5 API requests in parallel which produced match scores. Also I didn't user a template explaining what the model was. It was always a simple scoring prompt as a new conversation. Please score the probability (between 0 and 1) that the text .... is related to each of these topics. Please reply in JSON without any other explanation. the JSON template should look like this. {"conversational: 0.4, "vision": 0.3, "relocation" 0.5", "excursion": 0.1, "hunger": 0.2, "home_automation" 0.6 ....}. here's a list of keys: .... (about 50 of them) Each of the returned keys/values were then used to sent a follow up API request to determine which of the second stages to select such as turning on lights, replying with a joke or returning my appointments or TV listings etc. Then it was just a case of tuning the averaged thresholds to acceptable levels - this was also an intent as it took a while: buff or nerf ... intent by 0.2. Anyway maybe some of this might help you for the next attempt.

[@foddermike]

One way to solve the issue of commands firing off in other rooms/locations would be to have a listening device in each room, and the loudest location is where the command should be enabled. Unless, of course, an alternate location is specified.

[@technithusiast]

I like your creativity😁. loudness may backfire with kids involved 😆

[@jjbankert]

Practical question: how do you parse the chatgpt responses? I sometimes run into the problem that chatgpt returns the expected response + some natural text. And then that needs to be filtered out or it will lead to parsing errors.

[@technithusiast]

Good question. I ran into a similar issue as well. Since I’m expecting the result to be in the form of a JSON object, I scan the response for the first open curly brace and delete everything before it. So far this approach has been working but it’s not perfect and would fail if GPT added additional text after the end of the expected JSON response. Lemme know if you need more details and I can provide you with code snippets.

[@TobiasWeg]

I think the beginning of the Video was captivating, and I would love to see a short film of your experience, maybe make a play through of some of the things told about. Anyway, cool video, did you try to pull the temperature of ChatGPT to Zero? I did not see your other videos from before so maybe you did, but as general rule if you always want the same output Temperature zero is one important parameter.

[@technithusiast]

Thanks! Im not around my computer now so I don’t remember exactly but knowing me it’s probably between .5 - .75 the premise is that I wanted the AI to be a little more “human” by allowing variance in its response but create some consistency by telling it to output the answer in a specific format.

[@JanRademan]

This highlights an interesting paradox. AI is a great tool, because it will do things we don't expect, but conversely, we want it to reliably preform predictable tasks in a consistent manner. Which it can't because its core strength is doing unpredictable things.

[@technithusiast]

Bingo

[@StephanBuchin]

Excellent video clearly explaining the current limitations of this type of AI 🙂

[@technithusiast]

Thanks! I appreciate it!

[@TheRealDanielsan]

What version of GPT did you use? (3.5? or 4?) - also, did you explicitly give it all your entities in your pre-prompt? and finally, what was it's temperature setting?

[@technithusiast]

At the time of the video I use 3.5 and the temp was .7 I did send a list of entities with the payload but I limited to entities I can take action under the system

[@cleverman383]

I'd be curious to know what GPT would say about situations such as turning on the wrong lights if you told it to explain its rationale in detail before giving any commands

[@AntwanG]

When you told GPT that you wanted a PB&J Sandmich and it turned on the living room lights, you should have complied and ate in the living room. j/k liked and subscribed.

[@technithusiast]

Lolol you’re right 😆😆

[@LonesomeSoliloquy]

With the right caging of gpt through ke, I think it's possible that it could be very helpful. For example, direct it to turn on kitchen lights when you say you want to eat a sandwich. If you say turn on the lights, direct it to turn the lights on in the room that you're talking to it in by placing speakers and mics in each room. And always knowing the intent is definitely worth the extra hassle, at least until we understand these language models a bit more and can fine tune private versions a bit more.

[@technithusiast]

Definitely there’s some fine tuning I could do (like train the model to my specifications) but that’s a bit over-kill for what I’m going for. I’ll keep playing around with it and let everyone know what I find😬

[@StoutProper]

Having voice commands for lights is utterly pointless. It takes far longer to give the voice command and wait for the response than it does to just turn the lights on yourself, never mind so the extra power you’re consuming. A real smart home system would already know if you wanted the lights on, ie it’s dark and you’ve entered the room, so it turns them on. You shouldn’t need to be telling it to turn lights on and off all the time, it’s a complete waste of time and energy and will get tiresome very quickly

[@RavenWT]

That's an amazing experience ! I was wondering what was possible in term of automations, using OpenAI API and you seem to have pushed this really far! I'll check the video you mentionned with great interest! Thank you a lot! And keep going! Can't wait for next ones!

[@tensiondriven]

Fantastic vid, glad I found you. I think there are a handful of us out there trying to do similar things

[@technithusiast]

Welcome aboard!

[@asocialconsciousness8535]

Seriously cool video!! I have been wanting to integrate GPT into my home assistant in a more meaningful way for awhile now. I would love to have it set up as a voice assistant that can control some basic things but you took it to a whole new level! epic lol!! I would love to see more home assistant videos involving gpt!

[@technithusiast]

Glad you like it! Im currently starting to refine it further but stay tuned!

[@Fuego958]

This experiment is like BASE jumping into a tree line with a squirrel suit and a jet pack. Breathtaking.

[@s.patrickmarino7289]

I am assuming you were using GPT 3.5. How much context space does that give you? One of the things you could do would be to turn the guard rails back on and when it wanted to do something wrong, explain why that would be bad. The problem with closed LLMs is, you can't open them up for fine tuning. When you have limited Context space and you keep piling on rules, it will eventually start forgetting the first ones you gave it. One solution that I have not yet tried myself, use more than one instance of GPT and give each instance specific tasks. That gives you much more context space. Even if you refresh the context every morning, you will eventually run into issues I would think.

[@technithusiast]

Yup using 3.5. I tried a variety of things (that I didnt show in the video) One of the things that worked the best (similar to your suggestion) was splitting the commands down into smaller more straightforward parts this allowed me to do some level of error correcting but this had two significant drawbacks: - since GPT can’t “Remember” between requests, splitting it up increased accuracy but burns through a lot of tokens Since I have to keep passing the context. - Latentcy! The system got really slow to do anything that requires GPT to interpret. I fixed it by making parallel calls but compared to the version I currently have it was still noticeably slower where it felt like HA crashed when in reality it was latency.

[@Corbald]

Sounds like your experiment was a wild success! I'm also learning a lot from working on Athena. I've mostly solved the reliability issues you've experienced as a result of a rather complicated 'Prompt Structuring Algorithm' and a now deep understanding of how the Prompt effects the model's 'mind'. Additional things like chain-of-thought prompting and (hopefully soon!) tree-of-thought schema will absolutely up your response game! The only limitation I've discovered in how 'smart' these things can get is token costs and limits. I blew through $25 (I have a cap at $100, just in case) on the GPT-4_32k model over only 8 hours, during the _replacement_ of GPT-3.5... I'm getting a new PC in July, on which I should be able to host a copy of Vicuna or Wizard, so that's great, but GPT-4 is what really makes the intelligent behavior shine. 3.5 just can't seem to handle the size of prompts that are needed to get it working, and since I discovered that both model bias new (i.e. lowest text in the prompt) rather heavily, most of the 'Instruction' set actually needs to come AFTER the chat history and memory text. Once I've got GPT-4 fully integrated, I'm going to run my own crazy experiment, wherein I give GPT access to designing _it's own prompt structure_ in real-time, so it can redesign it's own mind! I fully expect it to crash and burn almost immediately, but if it doesn't, and the change/s it makes lead to it working _better_ then I'm in for a wild ride! Other things I'm working on: -I want to give her the ability to *both* write and run real-time python code, like we do with the command level interpreter, and I'd like to give her the ability to write and execute files, as well as some form of persistence between sessions. I'd be nice if the same variables are set in the environment as they were when she last ran code in it, even if I've reset her inbetween. -I'm searching for a STT engine that can differentiate speakers in a noisy area, which I know is a long standing issues in STT, but if she could identify and listen to specific speakers despite crosstalk, that'd be amazing! -Interaction is currently via a jupyter notebook's 'input' box, as displayed by my VSCode... I need something better, so she'll need to be in a real .py file so I can get something cooked up. Anyway, I may be able to help you get your home assistant listening to (and thinking with) Reason, but the TL;DR is basically chain-of-thought + us the LLM to check the LLM. Token costs get crazy (especially for 4, which is 10x more expensive, but also 4x bigger input!) but _quality_ improves greatly!

[@technithusiast]

Wow you seem very knowledgeable about this and have first-hand experience too! You should make a video to teach what you know, it would help people like me create better experiments!

[@Corbald]

@@technithusiast Lol, nope! I'm an explorer, like you! Just have a lot of time on my hands as of late. I'm the same guy you spoke with on another your video's comments, who was making suggestions about the prompting structure. So far, having implemented much of those suggestions myself, prompt structuring is like 90% of getting these things to perform well. Take a look at that "Tree of thought" prompting structure... it takes a bit of work to make it function, but it's amazing when it's working! Might solve your issues, at the cost of more tokens :/

[@technithusiast]

@@Corbald Will do! 👌🏾

[@joostdenboer5689]

Cool setup. Great findings. My house is also mostly controllable and I have a bunch of automations, but I’m missing the ‘smart’ part of a smart home. I have been thinking about AI to use it. Cool to see you we’re able to accomplish it. Is not the main problem the AI misses ‘context’ ? It should have known you’re daughter was asleep and therefore not turn on her light. I haven’t looked into it yet so I don’t have an idea how to do this. I do know I would not want to use ChatGPT but would rather use one of the upcoming opensource models like Alpaca, Vicuña, Falco etc and preferably something I could run locally as well. Looking forward what you’re going to do next (for inspiration :-))

[@technithusiast]

Yeah, context is part of the problem and there seems to be a subtle complexity to it that would be difficult to overcome without retraining the model with my family’s specific patters, behaviors, and preferences. In the future I do plan on using a local LLM to govern my smart home but they all seem volatile and unpolished and subject to heavy changes so I’m waiting till the dust settles

[@RandallReviere]

One idea is to leverage the fact that ChatGTP can be its own best critic... so at the end of every 'action chain', have it ask itself 'is the last thing I produced relevant?' And if the answer is no, then do nothing. Having said that, I think you are really getting somewhere. This is a great project... keep it up and keep evolving it. Very interesting stuff.

[@Laguy211]

That is quite the experiment you ran here and i will certainly go back to see the rest.

[@81-jdowlwp]

how about breaking the intents down into categories? does this require changes to: - car - house - alarms / reminders - ... then when chosen go one layer deeper

[@technithusiast]

So I’m currently testing this idea out. The biggest complaint I have with this method is that it makes the code and requests complicated :/

[@elfkind5590]

That moment when a coder does more for your marriage than a $400/hr therapist

[@technithusiast]

😮

[@bahramboutorabi5971]

Great job. You have an amazing mannerism that enriches your videos. Thank you.

[@technithusiast]

I appreciate that!

[@salah2fois]

I think the main problem here is giving the RIGHT context to chatGPT. Basically a well coded RAG with some prompt engineering will mitigate the uncertainty of the AI. It is worth the try.

[@jayong2370]

You read my mind or the KZbin algorithms read my mind 😮. This is exactly what I was thinking about today and you’ve started testing the ideas. Great video! Here’s and idea. Would you consider setting up CCTV cameras in your house to provide GPT more special awareness. I would love to chat more with you and brainstorm some ideas. 🙂

[@technithusiast]

I’m glad you’re enjoying the content!! Definitely more to come.

[@fr1sket363]

I wish everyone watched this video, it's a super balanced and unbiased view of AI autonomy. I think people forget there are sometimes more than one right answer to a question or even wrong answers. Your point of training from a large pool of essentially different view points both correct and incorrect ones, means exactly what you experienced, different outcomes to the same intent and or a different interpretation of your intent, that's unpredictable. I think no matter how good AI gets, it should never be unsupervised. This is just my personal opinion.

[@TheGoodContent37]

I'll share with you all an idea that I have had for years but I live in a third world country and I'm poor so I think I will never get to develop it, but maybe this is a good idea that someone else could use or see value in. I don't know if this idea is old or someone else already had it. It would be interesting to design a system that acts according to the sounds of the house. I thought about this thanks to my dog. Humans live with patterns, we almost always do the same things in our daily life, and that produces the same sounds. So, when I getting ready to go out the house I always take my wallet, phone, keys and my dog notices all that and gets ready to go out. I think he recognizes the sounds because he doesn't see me. BUT when I'm ready to take him to the vet I do different sounds and he inmediately acts all shy and hides, he knows he is going to the vet all by pure sound patterns. So I thought, wouldn't it be nice to have an ALEXA system that listens to all those personal sound patterns and gets ready to act when it senses I'll need something? For example, it senses I will go out, then it checks stove handles or turns off AC or things like that. Like a virtual dog that gets ready for all those patterns and gets ready to assist in any way you'll like it, even without the need to talk, the system could truly learn when those sound patterns mean that I will act in a certain way. Even walking over certain floor places emits different sounds, or the steps I take to do something. We live in a gaseous world, sound is everywhere around us so machines could learn those sounds and act accordingly (I'm a writter and I already wrote a few scripts with this idea in mind, I just hope AI develops enough to the point where I can produce those ideas instead of seeing rich people in first world countries having the same ideas and develping them years after I thought about them). I hope this is useful for someone. I don't usually share my ideas but I wanted to share this one because it's similar to the things said in this video and to at least leave something in the world before I die poor, alone and unknown.

[@bgone5520]

Interesting. Wondering which programming language you used for this.

[@technithusiast]

JavaScript :)

[@Jacobhopkins117]

That intro was freaking cool, man!

[@technithusiast]

Thanks im glad you enjoyed it!!

[@wild.legend-music]

A question popped in my head.. How do you expect it to know which lights to run on, if it doesn’t know where you are? (Note, I haven’t watched the other videos, but wanted to see what this is about) I’ve been thinking of something similar, like place 4 microphones in each corner of each room and just compare the which mics picks more sound when you say a command. That way it will know your exact location in the house and you don’t really need your phone all the time. It’s a bit of a privacy concern, but if you manage to make it secure so that no sound leaks out of your home server, you’ll be fine. Maybe use 2 servers, one connected to the internet for finding info and one offline that works with the mics and commands. Also you could use motion detection sensors, but the mics idea is better, because you can have multiple people in different rooms use commands and it will always know where the commands are coming from. I like the idea, but without it knowing your physical location in your house, I don’t think it can fully do what you really want. The microphones are the most ‘free’ and cheapest way to do it. Otherwise you can use cameras, which is a big nono, or wear some location tracking device, but then you have to wear it all the time. If you know a better way, I would like to hear about it. Thanks for the video, it was interesting and I will come back to watch the rest.

[@CherubEros]

15:58 this seems like the biggest hurdle. AI that can create images, video, sounds, voices, likenesses are terrifying not because it can, but because foes can use them to commit a crime with your identity, or falsify a crime that didn’t even happen. AI is great, people aren’t always stellar.

[@tutacat]

You need to use finetuning if you will use an LLM to run NLP so it is faster and better at focusing on your system and the things you want to do. You should also pass the context and response from GPT to double check itself, this double-inference has great results because it doesn't just keep going, but has a second run that looks for mistakes.

[@marsinsmusic]

Amazing video! Hallucination seems to be the next problem to resolve, but we have not yet resolved it in the human psyche.. That would be great if you would explore this specific topic. Thanks for the experience, very interesting. Looking forward to the next one (;

[@andrewferguson6901]

Pro tip. Hallucination is aberration of sensory apparatus. If the ai were hallucinating, it would be misreading your prompt or hearing prompts that don't exist. This phenomenon should be called something other than hallucination lest we lead ourselves astray looking for solutions

[@marsinsmusic]

@@andrewferguson6901 Agreed! Very good point here - thanks. What about calling it what we call for humans: lying-with-self_conscious_ignorance?

[@marsinsmusic]

aka Pretending-to-Know!

[@stereothrilla8374]

I love that you basically turned your house into Hal just for kicks!😂😂😂 Hope your house doesn’t have an airlock.

[@technithusiast]

😆😆

[@Modus_Pwnin]

This series is awesome. I wonder if you use mm wave with esp bluetooth tracking to notify ke of your location in the house if it would do a better job? Although with multiple people in the house maybe you would need a biometric voice recognition to know exactly who was talking to ke

[@technithusiast]

I already experimented with that idea :) check out these two videos: - kzbin.infoxdL3j7x-NBM - kzbin.infoPmM3rIJ6zzE Ke is able to use the history data from the sensors to answer and complete location based commands

[@davidatkinson9780]

This is the first of your videos that I’ve watched. I really appreciate the depth of thought you put into the KE/ChatGPT project, not to mention the technical chops. I’m retired now, one of the AI graybeards after 40 years. You are right on target to focus on “intent” understanding, but the larger issue is one of synchrony or “shared awareness “ of the AI agent KE and you. I suggest the agent has an incomplete model of you and context. This is because it “knows” nothing. ChatGPT is purely associative. There is no reasoning, no logic, no common sense, and no goal-directed behavior. Look at symbolic AI for this technology. It works, we just don’t call it AI anymore 😂. Anyway, really great. I’m a fan now and looking forward to having my mind blown!

[@technithusiast]

Very interesting concept around Synchrony. I later tried this experiment again with more context such as location and it did perform a lot better. 👌🏾

[@johnwilson5980]

I have been fascinated with the idea of having control of my air conditioning, so that I can not just set a temperature that I want to have in the room, but instead to be able to tell it that in summer I want to experience 21C, but because I have to keep my expenses down I want it to do it's best at achieving that goal but it must do so for less than x amount of dollars per week [KW/H per week]. I would want it to know how much I pay per KW/H and work with that and learn from experience what level of insulation my home has and take that into account. I have never seen a system that could do this. I would also like to be told how much I have saved to this frugality. This way I can have otal control over my comfort without getting bills I can't afford to pay.

[@technithusiast]

I think this would be a fantastic application of AI I believe the first useful application. Of AI in smart home tech that people will easily adopt would be analytics and predictions.

[@Syeleiswatching]

I think along with intent checking, it needs preference checking for household members. Something like, 'when judging an intent then prefer my preference'. This way when things are wierd, you have a way to say "not like that, please"

[@timothymaggenti717]

You are brave, I would not trust the tech that far just yet. But man it would be sweet if it was like in the movie's. Thanks

[@technithusiast]

I definitely understand people’s apprehension and they have good reason to be. With every technological push forward someone has to take those calculated risks.

[@milohobo9186]

I would like to imagine a future where ke gets a work-from-home job to start paying for upgrades to the home and becomes a great roommate.

[@technithusiast]

😆😆

[@Bennevisie]

You need to build in a feedback/weighting/preference mechanism as well as a context engine. Preference engine can confirm actions before taking it, and calibrate responses based on previous feedback. Context engine must take into consideration the current time, weather and temperature, your calendar and appointments, wearable / biometric data etc etc, to better infer intent.

[@PaulBrunt]

I've been getting good results with gtp3.5 still a work in progress. I've found that if you just get it to do "thinking" before outputting any action it seems to work fine the vast majority of the time,. The prompt I've currently put together for the pre action part is "Given the user input 'It's getting dark in here' do you need more information to update the json as desired? Explain your reasoning in detail then criticise your reasoning. Err on the side of caution and decide you need more information if the desired output is at all ambiguous." then you follow up its response by asking it to output the final action. I've also been playing with giving it a bit of personality in the response. A great one for the "It's getting dark in here" command was "Oh, it's getting dark, huh? How about being a bit more specific? Which room would you like me to enlighten? Or are you just enjoying the darkness? Feel free to let me know" The future is going to be like living in a 90s scifi sitcom 🙂

[@technithusiast]

Some of the considerations you pose are some that i discuss in my previous video: kzbin.info/www/bejne/q3emn6iAat1pq5Y Though I find asking GPT a series of questions to ensure stability comes at the cost of tokens. One of the challenges I gave was getting the most consistent and accurate response consuming the least amount of tokens

[@aaroncrawford9575]

First off, that was freaking cool. Thanks for sharing. Now for the brilliant idea from a noob. A literal noob. I’ve had chat GPT for about two weeks. That alone impresses me. Big idea: what if you let ke still run but give it the ability to learn the right response. Every time it completes an action based on one of your statements, have it send you a text asking if it was correct in performing the action. It might take a month but I bet you have a true smart home by the end of the learning period. Also consider adding hard rules like not turning on lights in certain rooms at certain times. Or only turning those lights on when you specifically ask to. Again I’m a noob that isn’t even sure what I suggested is even possible so don’t rip me apart if I said something stupid

[@technithusiast]

Hey thanks for watching and commenting. You’re not alone with your idea and based on some of the updates I’ve seen with openai we may be able to do something like that in the future.

[@scottgardener]

I am less worried about A.I. being The Terminator and more concerned about it being the broomsticks from Fantasia.

[@technithusiast]

😆😆

[@toleyk]

Pretty cool. I think you needed the guard rails on long enough, and memory for gpt to learn the correct way to interpret your intentions. Perhaps ask for clarification it confidence in the expected action wasn't high enough.

[@hvanmegen]

Just found your channel and I'm blown away by the quality of the content! You need more traction! Lets boost you to the moon!!!!!11

[@technithusiast]

Hopefully more people will put me on😁

[@skylineuk1485]

I have been working a lot with prompt engineering and probably most of those issues could be solved with gpt 4, asking it to act as an expert (appears to filter out the duff stuff somewhat), give it a house plan, what appliances you have and where etc. you are basically programming it in a sense to get the best from it. How long is the same context window left open also?

[@shynrou2]

Yeah I think so, too. The exact phrasing of the prompt is super important. Also to fix the randomness issue he could lower the temperature of GPT so it behaves more conservatively and predictable

[@peter.dolkens]

Would love you to expand on this experiment with GPT 4 turbo (increased context) and do some self-distillation to improve the accuracy of the intents. Ideally you'd also be using smart-speakers that can feed in which room you're in when you're asking the question, to help improve accuracy of things like "turn on the lights"

[@technithusiast]

You should check out the follow-up video if you haven’t seen it yet. kzbin.info/www/bejne/aoCppWNvmNqChbM

[@technithusiast]

This short may also be of interest if you haven’t seen it: I took your advice from the comments And it worked! kzbin.info/www/bejne/aovbhqaCo6mdiqM #smarthome #gpt4 kzbin.infoPmM3rIJ6zzE?feature=share

[@peter.dolkens]

Nice didn't see the short yet, but I definitely planned to try your how to video next. Heading off grid, so looking for something slightly more reliable and less reliant on internet than Siri. Currently working on getting the local transformer models tested and evaluated before integrating into HA, but your videos are bookmarked for reference!

[@Smirk_Station]

For a first run... amazingly pull off. 👌

[@technithusiast]

🙏🏾

[@Smirk_Station]

@@technithusiast 🙏

[@tutacat]

You can use ypur responses to requests for access, to finetune a model to respond how you like. You can probably also use a vector database "memory" and maybe a notes file to improve it over time.

[@NeoKailthas]

You need: 1.gpt4 2.Better prompt like asking it to review its own output before implementing changes. 3. More input, like your location.

[@B_MoreJ]

For once, the algorithm finally gets me to something I ACTUALLY want to watch. You have my sub sir.

[@ArnoSelhorst]

"Guardrails": Can't these be used in a first phase to train/fine-tune Ke? You could create place to store your decisions which Ke could then use to adapt its reactions to.

[@nonamemcgee4842]

That intro was amazing!

[@technithusiast]

Trying to keep things entertaining 😁

[@blackshard641]

Considering that most video game studios spend months debugging their games prior to release due to all the unexpected behaviors that arise from interacting subsystems, is it any wonder that dense machine learning models would have a few crossed wires that make them act peculiar the more we push them past their testing parameters?

[@metaimago]

Great video! New viewer here, instant subscriber. Really enjoyed the story and keen to see more of your content.

[@technithusiast]

Welcome aboard!

[@riflebird4842]

Wonderful experiment. I would suggest you to go to any one of the open source models and optimise it yourself for controlling a smart home. I think it would solve the problem. But like you said, even ML researchers don't know how GPT models are going to react. People did find out some weird behaviours, sometimes specific prompts unlocks whole new abilities, different behaviours and attitudes.

[@technithusiast]

The solution I have is pretty flexible as I’m able to swap out the LLMs. I’ll probably experiment with some different models in the future 👌🏾

[@riflebird4842]

@@technithusiast cool. Checkout huggingface for open source models. Waiting for your next update.