GPT TOOK OVER MY HOME - I learned why it's SCARY | | Chapter 4

I gave GPT full control over my house by integrating it into Home Assistant but then I had to shut it down! In this video, I'll go over how I gave GPT this power and what happened while it was in control.
This is a fascinating experiment and great conclusion to the the AI Master Class series.
#homeassistant #smarthome #chatgpt #homeautomation #masterclass
Enjoying the content? Show some love!
bmc.link/mmont...
Import Automations with this Chaperone Guide!
montaque22.git...
Music IDs:
- 4RVJLEOJH3O6CSJK
- 1CJQHXVGKKVPVH9X
Gear I used for this video
- Sony A7S iii: www.amazon.com...
- Aputure Light Dome: www.amazon.com...
- Aputure Amaran 200X S: www.amazon.com...
- Neewer Heavy Duty Light Stand: www.amazon.com...
DISCLAIMER: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with dope free content!

[@AshleyGittins]

Kinda blown away by the production values and storytelling here, I'm loving what you're doing!

[@technithusiast]

Thanks i appreciate it!

[@roberthentosh5635]

@@technithusiast I just watched the first 3:00 mins of the video and was about to say the same. Really done well. How many people you have on your production team!?

[@technithusiast]

@@roberthentosh5635 lololol it’s just me 😅

[@TheMechanic9143]

Give us more. This series is the most unique home assistant integration online

[@technithusiast]

Thanks I’m glad you like it!

[@StoutProper]

@@technithusiasthave you tried the new function call api? You can write and pass functions to gpt now to program exact responses from it. I think this will help enormously

[@technithusiast]

@@StoutProper yes! I find it very fascinating. I’m currently trying to find an easy way to incorporate it into home assistant’s automations

[@realtimestatic]

Honestly a smart home with a personal assistant based on LLMs with guardrails is something I've been looking forward to the moment LLM's became better!

[@technithusiast]

Same here!!

[@nightynight5990]

I think it might help to tell GPT about the layout of the house and what each room is used for. As well as telling it that you dont have an AC unit, and what it should do if it cannot find anything to help. Maybe that would make it work better.

[@dimitriosmolfetas4711]

Great idea and could also add a node that checks for a presence through a presence sensor, so it would turn on the lights if a person is present and not laying down for example

[@ColinTimmins]

I agree, it needs to know more information. Build a list of every room, everything in it. User can add and remove items off the list.

[@vikasreddy7015]

this would be the first step to be done, not sure if this guy has done it.

[@oneman5753]

Does this defeat the purpose to some degree? Almost sounds like you want to build a house object with all the stuff you want gpt to control and specify what it should do if it can't find things, but at that point are you defeating the point of using gpt in the first place? I honestly don't know i haven't played with it enough i'm all conceptual at this point

[@Magnetar_StarHeart]

Ya damn straight it was. Ha. Golden. ☆♡

[@brian2590]

You have courage! This is where Open Source models and self hosted IoT comes into play for me. Home Assistant tasks do not need a super intelligence or model that can do 1000 circus tricks. We will soon have smaller self hosted models that can eliminate some of ChatGPT from the equation. Exciting times!

[@traxeonic3600]

Your perspective and experience with the ChatGPT integration is exactly the perspective people need to see. The comment on the fact that untrue data was used in training is a key point many must need to know.

[@technithusiast]

Thanks, and im glad you appreciate the content! I think we can learn a lot by pushing tech to the extremes cuz it often reveals interesting insights the spurs new ideas, new safety protocols, and new technology

[@Ewoodster]

Why did I just find this channel now. This is next level stuff! Outstanding.

[@technithusiast]

Glad you like it!

[@SimplyElectronicsOfficial]

Dude, you need to use a fine-tuned model. Without Fine-tuning you will never know what the model will return. GPT Temperature should also be set to 0.

[@technithusiast]

You’re absolutely correct. I was hoping the model right out of the box would serve my purpose but I agree that training it would yield significantly improved results. The only caveat I see is that this may not scale well. It would be nice if HA provided a way to train models off the automations we create

[@OldManShoutsAtClouds]

​@technithusiast that last part is your job 😉

[@Caffin8tor]

It's these kind of unexpected emergent behaviors along with the fact that AI is now capable of providing feedback into its own development that I think something huge will happen with AI before this year is over.

[@Holphana]

I hope so! 🎉

[@theloniousMac]

Something huge happens every day.

[@TheGrobe]

I definitely don’t hope so! I think we have no idea what we are screwing around with

[@Vamplord111]

That would be awesome if it happened that soon 😊

[@techinvestor9443]

Yeah, literally, could happen tomorrow. 100% agree.

[@danlindy9670]

Thank you! Absolutely fantastic that you did this experiment and shared the result with the world, demonstrating the fact that LLM’s are not the kind of predictable tech that we’re used to. This new world is more wild and more dangerous. We are going to have to learn quickly how to survive (and hopefully thrive) in it.

[@arianaponytail]

love to see more of a system like this in action. not just you telling what happend.

[@rasbe6863]

I love this series in this content. I am in to text to speech and voice alerts in my home. I am waiting until I can create local controlled speakers to implement something like this. But I will not allow skynet to control my world. So I'm waiting for you to figure it all out and then I'll just copy you. LOL

[@technithusiast]

Hahaha 🤣 🤣🤣 I don’t mind being the guinea pig

[@UliTroyo]

This was a lot of fun! Thank you for putting yourself in danger for the rest of us ;D

[@technithusiast]

😆

[@johnhartman718]

Nice! I found this to be one of the more interesting rabbit holes on KZbin. Just start down the Smart Home world and look forward to your content ! P.S. don't stop the GPT

[@technithusiast]

Thank you!

[@ColinTimmins]

Dive deep man. I am programming my own system. I am so freaked hyped! I, can not write CODE. I am too dyslexic, too slow. So far I have it “remove” or “translate”, and I can one shot any basic command. I have worked like a madman the last three months and boy have I learned so much. It’s an exciting time to be alive! 😊

[@johnblack9499]

Man, what an awesome video :-) I can watch 'brainiacs' talking about the future of AI all over YT, but here you are actually putting it into practice. Having dabbled with my own AI, I appreciate all the effort you must be putting into this and then taking the time out to share your findings with us. I do wonder what your family thinks of all of this :-) good luck, I shall watch future videos with deep interest, cheers from a fellow AI enthusiast. PS I do like your 'wild animal' analogy, it does seem like we desperately trying to tame a newly discovered animal that may well be more powerful than us.

[@marissya7837]

You’re very engaging to listen to. Keep ‘em coming! I’m not an engineer but am actively learning process and appreciate videos like these.

[@mtnsolutions]

Very cool. What a fun experiment. I bet if it were trained a bit more, KE could be a real home assistant

[@technithusiast]

I agree! I was hoping that GPT-3 out-of-the-box would work but using it as a virtual assistant seems to warrant training data to fine-tune the responses.

[@wagnerfontes2]

@@technithusiast I was thinking just that! Maybe if you gave KE a little more specific training, like follow-up questions or instructions (e.g. Why did you turn that light on? I'd rather stay in the kitchen) it might become more fitted to your expectations. I wish I had time and expertise to implement something similar...

[@tricilin]

Yes! If you are up for it a "tutorial" would be very welcome. This was the first video I've seen from you and I immediately subscribed! Keep up the amazing work and have fun.

[@johnwheatley231]

I use a prompting system to get gpt 4 in the playground to self check its own logic and revise its end output based on this self assessment. It makes the outputs way more reliable and eliminates most if not all hallucinations. Here's how it works: In the system box put this: You are a group of 3 individuals who collaborate on producing the best results for user prompting. Person 1 is named John. John is scientific and factual in his responses. Person 2 is named frank. Frank answers based on intuition and is more in tune with emotional and social cues. Person 3 is named Joe. Joe assesses both John's and Frank's answers and combines them into a balanced answer including parts from both answers. John and Frank then critique Joe's compiled answer using different arguments. John's critique serves as fact checking while Frank's critique challenges the interpretation of the prompt. Joe then revises his answer using his own compiled answer and the critiques of John and Frank and incorporates everything into a final answer. Show all answers and thought processes involved from John, Frank and Joe. Probably overkill for what your trying to do but the approach might be useful. This works a charm for any of the difficult prompting problems I've tried. Funny enough I posted this on Reddit and my account now shows that I have no posts or comments even though I have 99 karma and a stary award from this post...hmmm...

[@tuxuhds6955]

Not a wild beast - You've just described my two years old inquisitive kid. She wanted to eat so it made sense to her to get the kitchen stool. At first we didn't want her picking that up for safety reasons but then I got curious... LSS She wanted to climb the stool up to the sink so she could wash her favorite plate, for dinner. Not a wild beast, a toddler.

[@nathanielalderson9111]

And what happens when that "toddler" has the keys to the gun cabinet? Or wants to play with knives? Or gets angry and thinks using fire to make an annoying curtain disappear is a good idea? The toddler is a good analogy. There's a reason why there's an age of accountability in humans. Computation process have no such milestone, no safeguard, no guardrails, no fallback, and no recourse for mistakes, deadly included.

[@technithusiast]

Yoooooo FACTS. 💯

[@StormyHotwolf88]

YESSS!!! I'm so excited on where you went with this! I really hope to see more development on this project!

[@technithusiast]

More to come!

[@JonathanYankovich]

Fantastic vid, glad I found you. I think there are a handful of us out there trying to do similar things

[@technithusiast]

Welcome aboard!

[@StephanBuchin]

Excellent video clearly explaining the current limitations of this type of AI 🙂

[@technithusiast]

Thanks! I appreciate it!

[@RavenWT]

That's an amazing experience ! I was wondering what was possible in term of automations, using OpenAI API and you seem to have pushed this really far! I'll check the video you mentionned with great interest! Thank you a lot! And keep going! Can't wait for next ones!

[@jeibar]

Mate you should work with the HA team and fully developed all this with them . They’re currently hiring people if I’m not mistaking . You’re flipping genius

[@technithusiast]

😆 I appreciate it and im glad you enjoyed the video! The thought has crossed my mind 👨🏾‍💻

[@andrewowens5653]

@Technithusiast. Some ideas you might try: since it's a home automation system and there's many people living in the home, you need to make the AI understand it has to process information from the point of view of all the occupants, including animals. You might want to try using a small language model with a large context length so that you could implement some more sophisticated rules. The system needs to know if they're conflicting commands between the occupants, and or other situational variables. It would be interesting if you also had cameras set up surrounding your house so that the house itself would know the external situation. Anyway, I enjoyed your video and I wish you luck.

[@Corbald]

Sounds like your experiment was a wild success! I'm also learning a lot from working on Athena. I've mostly solved the reliability issues you've experienced as a result of a rather complicated 'Prompt Structuring Algorithm' and a now deep understanding of how the Prompt effects the model's 'mind'. Additional things like chain-of-thought prompting and (hopefully soon!) tree-of-thought schema will absolutely up your response game! The only limitation I've discovered in how 'smart' these things can get is token costs and limits. I blew through $25 (I have a cap at $100, just in case) on the GPT-4_32k model over only 8 hours, during the _replacement_ of GPT-3.5... I'm getting a new PC in July, on which I should be able to host a copy of Vicuna or Wizard, so that's great, but GPT-4 is what really makes the intelligent behavior shine. 3.5 just can't seem to handle the size of prompts that are needed to get it working, and since I discovered that both model bias new (i.e. lowest text in the prompt) rather heavily, most of the 'Instruction' set actually needs to come AFTER the chat history and memory text. Once I've got GPT-4 fully integrated, I'm going to run my own crazy experiment, wherein I give GPT access to designing _it's own prompt structure_ in real-time, so it can redesign it's own mind! I fully expect it to crash and burn almost immediately, but if it doesn't, and the change/s it makes lead to it working _better_ then I'm in for a wild ride! Other things I'm working on: -I want to give her the ability to *both* write and run real-time python code, like we do with the command level interpreter, and I'd like to give her the ability to write and execute files, as well as some form of persistence between sessions. I'd be nice if the same variables are set in the environment as they were when she last ran code in it, even if I've reset her inbetween. -I'm searching for a STT engine that can differentiate speakers in a noisy area, which I know is a long standing issues in STT, but if she could identify and listen to specific speakers despite crosstalk, that'd be amazing! -Interaction is currently via a jupyter notebook's 'input' box, as displayed by my VSCode... I need something better, so she'll need to be in a real .py file so I can get something cooked up. Anyway, I may be able to help you get your home assistant listening to (and thinking with) Reason, but the TL;DR is basically chain-of-thought + us the LLM to check the LLM. Token costs get crazy (especially for 4, which is 10x more expensive, but also 4x bigger input!) but _quality_ improves greatly!

[@technithusiast]

Wow you seem very knowledgeable about this and have first-hand experience too! You should make a video to teach what you know, it would help people like me create better experiments!

[@Corbald]

@@technithusiast Lol, nope! I'm an explorer, like you! Just have a lot of time on my hands as of late. I'm the same guy you spoke with on another your video's comments, who was making suggestions about the prompting structure. So far, having implemented much of those suggestions myself, prompt structuring is like 90% of getting these things to perform well. Take a look at that "Tree of thought" prompting structure... it takes a bit of work to make it function, but it's amazing when it's working! Might solve your issues, at the cost of more tokens :/

[@technithusiast]

@@Corbald Will do! 👌🏾

[@asocialconsciousness8535]

Seriously cool video!! I have been wanting to integrate GPT into my home assistant in a more meaningful way for awhile now. I would love to have it set up as a voice assistant that can control some basic things but you took it to a whole new level! epic lol!! I would love to see more home assistant videos involving gpt!

[@technithusiast]

Glad you like it! Im currently starting to refine it further but stay tuned!

[@MasterBel2]

I see an interesting parallel between hint/guess-based communication; a) removing a necessary feedback loop between you & GPT, and b) requiring GPT to infer an intent for GPT’s action that you know, but you’ve chosen not to communicate directly. I’ve recently discovered there’s boundless joy in clear, purposeful communication and directly asking for things, rather than relying on the people around you to guess at your desires. Moreover, asking someone for something is way nicer than just telling them you want it.

[@technithusiast]

Communication is tricky thing since it’s so nuanced. You can communicate intent with simply your eyes and as humans we can pick up on things like that. It’s really refreshing when people are clear with their words and it’s natural for people to rely on context when we talk. For AI to become mainstream it’s going to have to become reasonably adept at it for wider adoption

[@patrickcollins4630]

Awesome production value in intro dude - had to re-watch it.

[@technithusiast]

Glad you liked it!

[@0bscura]

Fascinating work. GPT's ability to leverage it's language model is powerful, but it's ability to think on the fly is like that of a small child.

[@technithusiast]

Totally agree. It has its limitations but you can squeeze a lot of utility out of it if you can structure the commands right

[@TimMattison]

I did a project with my security cameras and GPT-4 and I found that the trick was to write the prompt and then break it up into smaller prompts you chain together if it has a hard time. Splitting my one giant prompt into three small prompts was not only more reliable but cost 100x less (!). There’s no one answer. Just a lot of fun experimenting. Love the video. Keep it up!

[@bahramboutorabi5971]

Great job. You have an amazing mannerism that enriches your videos. Thank you.

[@technithusiast]

I appreciate that!

[@DavonAllen92]

whats cool about this is that it goes to show you that communication and context is needed no matter if you are machine or human. it seems a lot like talking to a child who has the ability to do anything. but has no context or aware ness of the exact needs but instead of asking it just processed to do what you think it meant. those guard rails are intuitive because its how we mostly work as humans.

[@monas.6839]

Came here bc of your collab with Shane Whatley…the only bad thing about it is that I had not found you sooner…love your content!

[@technithusiast]

Welcome aboard!!

[@fanaticdavid]

What a great video! It looks like I may have to dive into your backlog of videos here on YT. Subscribed!

[@technithusiast]

Really glad you like it! I have a membership and discord too if you would like to join. You may find it valuable if you’re into building out automations and tinkering with smart-home tech: youtube.com/@technithusiast/join (the link works best from a browser)

[@davidatkinson9780]

This is the first of your videos that I’ve watched. I really appreciate the depth of thought you put into the KE/ChatGPT project, not to mention the technical chops. I’m retired now, one of the AI graybeards after 40 years. You are right on target to focus on “intent” understanding, but the larger issue is one of synchrony or “shared awareness “ of the AI agent KE and you. I suggest the agent has an incomplete model of you and context. This is because it “knows” nothing. ChatGPT is purely associative. There is no reasoning, no logic, no common sense, and no goal-directed behavior. Look at symbolic AI for this technology. It works, we just don’t call it AI anymore 😂. Anyway, really great. I’m a fan now and looking forward to having my mind blown!

[@technithusiast]

Very interesting concept around Synchrony. I later tried this experiment again with more context such as location and it did perform a lot better. 👌🏾

[@hvanmegen]

Just found your channel and I'm blown away by the quality of the content! You need more traction! Lets boost you to the moon!!!!!11

[@technithusiast]

Hopefully more people will put me on😁

[@marsinsmusic]

Amazing video! Hallucination seems to be the next problem to resolve, but we have not yet resolved it in the human psyche.. That would be great if you would explore this specific topic. Thanks for the experience, very interesting. Looking forward to the next one (;

[@andrewferguson6901]

Pro tip. Hallucination is aberration of sensory apparatus. If the ai were hallucinating, it would be misreading your prompt or hearing prompts that don't exist. This phenomenon should be called something other than hallucination lest we lead ourselves astray looking for solutions

[@marsinsmusic]

@@andrewferguson6901 Agreed! Very good point here - thanks. What about calling it what we call for humans: lying-with-self_conscious_ignorance?

[@marsinsmusic]

aka Pretending-to-Know!

[@stereothrilla8374]

I love that you basically turned your house into Hal just for kicks!😂😂😂 Hope your house doesn’t have an airlock.

[@technithusiast]

😆😆

[@metaimago]

Great video! New viewer here, instant subscriber. Really enjoyed the story and keen to see more of your content.

[@technithusiast]

Welcome aboard!

[@AntwanG]

When you told GPT that you wanted a PB&J Sandmich and it turned on the living room lights, you should have complied and ate in the living room. j/k liked and subscribed.

[@technithusiast]

Lolol you’re right 😆😆

[@DeruwynArchmage]

I think a few things might help: First, to help eliminate the “multiple personalities” problem, give it a personality that it should adopt. A good one might be Jarvis from the Avengers. Second, instead of an all or nothing approach, let it ask itself if it’s sure it knows what you want it to do. If it’s unsure, then have it ask. Also, explicitly tell it that “do nothing” is a valid option. Third, I don’t know if you’ve done this, but explicitly tell it to try and determine whether you’re talking to it or if you might be talking to yourself or someone else. Fourths, to help keep it grounded, make sure that there is a step where, after it has decided that you are asking it to do something, have it search for appropriate context in some files you have created, or perhaps that it has updated, and use that as grounding for what options it has. So you’d want to list all of the devices you have and actions it can take. Then, when it decides it should do something, it can pull out the relevant parts of those files and include them in the next prompt where it decides what to do. Let it have the option to take multiple actions if it wants. Fifth, have a file where you describe the occupants of your home. Let it also create a different file to see if it thinks there is a guest in the house. You should have it record when it last heard them talking and maybe the last few things they said so that it can figure out if they’ve left and it can move them to a history file instead of the guest file. Sixth, if you have multiple microphones that give it input and it can differentiate the source of the input, let it use that to try and determine the location of everyone it thinks is in the house. All of your smart devices should have their locations explicitly labeled in your device file. If it doesn’t know where someone is that it suspects is in the house is, have it produce its best guess or top N guesses. Make sure that one of the pieces of data that it always has is the current date and time. Have it record the past schedules of people in the home so that it can build up a library of what it thinks people are probably doing at any given timeframe. This should allow it to figure out that your child is probably asleep in their room. If it doesn’t have past history, have it give its best guess based on what it does know about the individual and people like them in general. So it should be able to guess that kids go to bed early. That kind of context should really help it do what it should. Seventh, have a step where it predicts what the expected consequences of each of its proposed actions will be. Tell it that the expected emotional states of the occupants is very important. Have it chase down the chain of expected events. So, light in the child’s bedroom turns on, the child wakes up, the child is upset, the child goes to get their parent, the parent is upset. Is this a better result than doing something else or nothing at all? If not, don’t do it. If it thinks it will make everyone happier, have it justify why it thinks so and then have it examine that chain of thought in a separate prompt as if someone else said it and have it judge if the beliefs are justified. This should mean that it doesn’t take actions that are likely to make anyone it knows about more unhappy. I think if you do all of that, you should get something pretty close to what you want. Use GPT3.5-turbo for the easier steps that you’ve verified that it’s pretty good at to speed up the results.

[@DeruwynArchmage]

Oh, and have it record a history of its past actions and have it check if that past action had a good or a bad outcome, so if you told it, “no, that’s not what I meant!” It should be able to learn from that and not do it again.

[@AnmAtAnm]

Sounds like your system could use a step to brainstorm possible intents (possibly annotated with pro/cons/assumptions/prereqs/expectations) and then critique/vote/stack-rank to help select the best match to human request. Similar from intent to resolving action. Between brainstorming and critique, you might have more formal validations: does this AC unit id exist, to filter out bad data. Are you planning to retrain a non-GPT LLM with all that success/failure data?

[@technithusiast]

I’ve had error correcting steps in some of my iterations but best case scenario if it chooses entities/commands that don’t exist it would simply say it can’t do anything about it or worst case go into an infinite loop of trying get the entity and command correct. Retraining would help the consistency but I need a scalable approach that’s easy to maintain. With some new features home assistant release in June release, a few ideas come to mind but it would take some time to setup.

[@StoutProper]

@@technithusiasta few suggestions for improvements: 1) use the gpt4 model 2) turn down it’s temperate setting and up its top_ p parameter, making it conservative and deterministic 3) use tree of thought reasoning, which will evaluate a number of different solutions to the problem and synthesise the best one from them 4) use huggingface’s multi agent mini AGI or audio gpt or something similar and create different agents for different things 5) train and fine tune the model using human feedback learning 6) look into creating a database that gives it long term memory and the ability to access previous chats and commands and intent resolutions etc (basically expanding the number of tokens of the chat context massively)

[@StoutProper]

@@technithusiastoh and implement voice recognition and possibly facial recognition too

[@jayong2370]

You read my mind or the KZbin algorithms read my mind 😮. This is exactly what I was thinking about today and you’ve started testing the ideas. Great video! Here’s and idea. Would you consider setting up CCTV cameras in your house to provide GPT more special awareness. I would love to chat more with you and brainstorm some ideas. 🙂

[@technithusiast]

I’m glad you’re enjoying the content!! Definitely more to come.

[@circuitguy9750]

Your production is next level. Awesome! What's the music by the way? My google-fu failed me on your "Music IDs".

[@technithusiast]

Hey thanks! The song is called "I like that" by Ivy States from SoundStripe l.linklyhq.com/l/1q8S1

[@icarusgaming6269]

KE trying to gaslight you into thinking it turned on an imaginary AC unit has to be the scariest response

[@technithusiast]

Definitely had me looking over my shoulder.

[@Taterxxwardy]

We want more in depth guides on how you accomplish this!

[@technithusiast]

Thanks for the feedback!! I’ll look into it 😬

[@SimplyElectronicsOfficial]

I have successfully built the foundations of a Python project which allows a user to make requests to GPT-4 to control their HomeAssistant instance entirely via the HA REST API. Still a lot of work to do but it works! Hail Function-Calling! There is literally nothing this can't do now. It has the capability to look at device states and attributes and call one or more pre-written functions to allow it to execute anything it believes it should do as per the users request. I haven't built automation into it yet. But I will experiment with it. Thumbs up if I should Commit to Github?

[@Sci-fi-Si]

Really interesting. Are you using Node Red to interface with ChatGPT? Looks like I'll have to look over your previous vids! Keep up the great work! :D

[@technithusiast]

Yes indeed. Since node red allows me to use JavaScript I can get away with a lot of creative things that YAML users would have a harder time doing. But to be fair im not that well-versed with yaml so im technically taking the “easy” way out 😅

[@foddermike]

One way to solve the issue of commands firing off in other rooms/locations would be to have a listening device in each room, and the loudest location is where the command should be enabled. Unless, of course, an alternate location is specified.

[@technithusiast]

I like your creativity😁. loudness may backfire with kids involved 😆

[@leonardodilucas235]

I like your channel buddy. Keep up the great work!! Hugs from Brazil 🎉

[@technithusiast]

Much appreciated!!

[@TobiasWeg]

I think the beginning of the Video was captivating, and I would love to see a short film of your experience, maybe make a play through of some of the things told about. Anyway, cool video, did you try to pull the temperature of ChatGPT to Zero? I did not see your other videos from before so maybe you did, but as general rule if you always want the same output Temperature zero is one important parameter.

[@technithusiast]

Thanks! Im not around my computer now so I don’t remember exactly but knowing me it’s probably between .5 - .75 the premise is that I wanted the AI to be a little more “human” by allowing variance in its response but create some consistency by telling it to output the answer in a specific format.

[@salah2fois]

I think the main problem here is giving the RIGHT context to chatGPT. Basically a well coded RAG with some prompt engineering will mitigate the uncertainty of the AI. It is worth the try.

[@twylxght]

For once, the algorithm finally gets me to something I ACTUALLY want to watch. You have my sub sir.

[@ShaneHerald]

wow thats pretty cool ..... amazing work man!!! you should definitely keep working on that ....

[@technithusiast]

Thanks will do!

[@Laguy211]

That is quite the experiment you ran here and i will certainly go back to see the rest.

[@joostdenboer5689]

Cool setup. Great findings. My house is also mostly controllable and I have a bunch of automations, but I’m missing the ‘smart’ part of a smart home. I have been thinking about AI to use it. Cool to see you we’re able to accomplish it. Is not the main problem the AI misses ‘context’ ? It should have known you’re daughter was asleep and therefore not turn on her light. I haven’t looked into it yet so I don’t have an idea how to do this. I do know I would not want to use ChatGPT but would rather use one of the upcoming opensource models like Alpaca, Vicuña, Falco etc and preferably something I could run locally as well. Looking forward what you’re going to do next (for inspiration :-))

[@technithusiast]

Yeah, context is part of the problem and there seems to be a subtle complexity to it that would be difficult to overcome without retraining the model with my family’s specific patters, behaviors, and preferences. In the future I do plan on using a local LLM to govern my smart home but they all seem volatile and unpolished and subject to heavy changes so I’m waiting till the dust settles

[@davidberry8463]

I just want to keep watching the brother program! Great nerd content!

[@technithusiast]

✊🏾💪🏾

[@peter.dolkens]

Would love you to expand on this experiment with GPT 4 turbo (increased context) and do some self-distillation to improve the accuracy of the intents. Ideally you'd also be using smart-speakers that can feed in which room you're in when you're asking the question, to help improve accuracy of things like "turn on the lights"

[@technithusiast]

You should check out the follow-up video if you haven’t seen it yet. kzbin.info/www/bejne/aoCppWNvmNqChbM

[@technithusiast]

This short may also be of interest if you haven’t seen it: I took your advice from the comments And it worked! kzbin.info/www/bejne/aovbhqaCo6mdiqM #smarthome #gpt4 kzbin.infoPmM3rIJ6zzE?feature=share

[@peter.dolkens]

Nice didn't see the short yet, but I definitely planned to try your how to video next. Heading off grid, so looking for something slightly more reliable and less reliant on internet than Siri. Currently working on getting the local transformer models tested and evaluated before integrating into HA, but your videos are bookmarked for reference!

[@smartduck904]

It would be cool if it slowly dimmed the lights upwards as your alarm goes off and it just slowly brought light into your room

[@aaroncrawford9575]

First off, that was freaking cool. Thanks for sharing. Now for the brilliant idea from a noob. A literal noob. I’ve had chat GPT for about two weeks. That alone impresses me. Big idea: what if you let ke still run but give it the ability to learn the right response. Every time it completes an action based on one of your statements, have it send you a text asking if it was correct in performing the action. It might take a month but I bet you have a true smart home by the end of the learning period. Also consider adding hard rules like not turning on lights in certain rooms at certain times. Or only turning those lights on when you specifically ask to. Again I’m a noob that isn’t even sure what I suggested is even possible so don’t rip me apart if I said something stupid

[@technithusiast]

Hey thanks for watching and commenting. You’re not alone with your idea and based on some of the updates I’ve seen with openai we may be able to do something like that in the future.

[@gnashermedia]

Wow. You just got my sub and like. What a great content. I would be definitly interessted in more.

[@technithusiast]

Welcome aboard!

[@milohobo9186]

I would like to imagine a future where ke gets a work-from-home job to start paying for upgrades to the home and becomes a great roommate.

[@technithusiast]

😆😆

[@killbotprime]

Badassery and brilliance in action earned a sub.

[@technithusiast]

I’m glad you like it!

[@Ticklestein]

Just a comment to make sure more people see this vid. Also liked and subbed. No need to even ask. Great video.

[@technithusiast]

I appreciate you 😁

[@Ticklestein]

@@technithusiast I also sent the video to a friend of mine who works in the field of AI. He's gonna love this. Keep messing around with this stuff. Maybe widen the guardrails in stead of removing them...Look at things like AgentGPT. You'll appreciate that one. It'll fully solve your intentions problem

[@technithusiast]

Noice! I’ll take a look!

[@Malhorne]

Excellent :o) Thank you... I am working on a project of "on site dedicated HOMEIA"... thinking at first it shall need to know only the occupants not ROW and keep track of appropriate scénarios. Thanks a lot again

[@technithusiast]

🙏🏾

[@JanRademan]

This highlights an interesting paradox. AI is a great tool, because it will do things we don't expect, but conversely, we want it to reliably preform predictable tasks in a consistent manner. Which it can't because its core strength is doing unpredictable things.

[@technithusiast]

Bingo

[@aidanb8719]

Came across this video and really enjoyed the way you described the rationale behind it all. Really would like to see how you got it all set up and running.

[@technithusiast]

Thanks!! Check out the other videos in my AI Masterclass playlist to see how this came together

[@aidanb8719]

@@technithusiast I have found it and will be following it. Decided to build a new Home Assistant as moving to the sky connect has messed things up a bit so will use Node Red more from seeing your work.

[@LonesomeSoliloquy]

With the right caging of gpt through ke, I think it's possible that it could be very helpful. For example, direct it to turn on kitchen lights when you say you want to eat a sandwich. If you say turn on the lights, direct it to turn the lights on in the room that you're talking to it in by placing speakers and mics in each room. And always knowing the intent is definitely worth the extra hassle, at least until we understand these language models a bit more and can fine tune private versions a bit more.

[@technithusiast]

Definitely there’s some fine tuning I could do (like train the model to my specifications) but that’s a bit over-kill for what I’m going for. I’ll keep playing around with it and let everyone know what I find😬

[@StoutProper]

Having voice commands for lights is utterly pointless. It takes far longer to give the voice command and wait for the response than it does to just turn the lights on yourself, never mind so the extra power you’re consuming. A real smart home system would already know if you wanted the lights on, ie it’s dark and you’ve entered the room, so it turns them on. You shouldn’t need to be telling it to turn lights on and off all the time, it’s a complete waste of time and energy and will get tiresome very quickly

[@nonamemcgee4842]

That intro was amazing!

[@technithusiast]

Trying to keep things entertaining 😁

[@CherubEros]

15:58 this seems like the biggest hurdle. AI that can create images, video, sounds, voices, likenesses are terrifying not because it can, but because foes can use them to commit a crime with your identity, or falsify a crime that didn’t even happen. AI is great, people aren’t always stellar.

[@AlexBenfica]

Excelente title. Saved to watch later.

[@technithusiast]

Thanks! 😁

[@VAFlash]

Subscribed. Excellent video. Great story telling!

[@technithusiast]

I appreciate it!

[@74Gee]

I did a similar project for fun and it worked really well. It was not about choosing from a list given an arbitrary prompt though. For me it was about removing every possibility until there's nothing left but the correct intents. One way to do this is by consensus. I used 5 API requests in parallel which produced match scores. Also I didn't user a template explaining what the model was. It was always a simple scoring prompt as a new conversation. Please score the probability (between 0 and 1) that the text .... is related to each of these topics. Please reply in JSON without any other explanation. the JSON template should look like this. {"conversational: 0.4, "vision": 0.3, "relocation" 0.5", "excursion": 0.1, "hunger": 0.2, "home_automation" 0.6 ....}. here's a list of keys: .... (about 50 of them) Each of the returned keys/values were then used to sent a follow up API request to determine which of the second stages to select such as turning on lights, replying with a joke or returning my appointments or TV listings etc. Then it was just a case of tuning the averaged thresholds to acceptable levels - this was also an intent as it took a while: buff or nerf ... intent by 0.2. Anyway maybe some of this might help you for the next attempt.

[@PhilThomas]

I am reminded of the service dog that failed training, it learned how to do the actions, but failed to relate them to their corresponding command... like thos dog would open the fridge, but only when he felt like it

[@technithusiast]

Yup sound just like GPT 😆

[@Mephmt]

"I just wanted to see what would happen if I gave it access to everything." "What would it do with no checks and balances?" You're a mad man! I love it. I would have done the same. This is why we are not in charge of AI R&D. Keep the content coming! This your videos, of which I binged all of them in one night once I discovered your channel, are truly inspiring to me! Great work!

[@technithusiast]

Lolol you so right. I’d have R&D underwater if they gave me keys 🤣🤣 So glad you’re enjoying the content!

[@Syeleiswatching]

I think along with intent checking, it needs preference checking for household members. Something like, 'when judging an intent then prefer my preference'. This way when things are wierd, you have a way to say "not like that, please"

[@Anthony-ys8sr]

I'd love to learn more about your automations 🙂 Please do another video about it 🙏

[@technithusiast]

Will do!

[@tutacat]

You need to use finetuning if you will use an LLM to run NLP so it is faster and better at focusing on your system and the things you want to do. You should also pass the context and response from GPT to double check itself, this double-inference has great results because it doesn't just keep going, but has a second run that looks for mistakes.

[@scottgardener]

I am less worried about A.I. being The Terminator and more concerned about it being the broomsticks from Fantasia.

[@technithusiast]

😆😆

[@tutacat]

You can use ypur responses to requests for access, to finetune a model to respond how you like. You can probably also use a vector database "memory" and maybe a notes file to improve it over time.

[@boukm3n]

*damn, this brotha is amazing* 💯🗿🗿🗿 I’m trying to integrate tree of thoughts into my project rn. Honestly would love to see you take a crack at it. Watching a few of your videos, you’d probably make crazy progress. You earned a new sub 🗿🗿🗿

[@technithusiast]

Thanks for the sub! I haven't heard of tree-of-thought until you and some of the other viewers mentioned it. I'll definitely take a look

[@Glebean]

Amazing content keep up the good work 👏👏👏

[@technithusiast]

Thank you!!

[@Jacobhopkins117]

That intro was freaking cool, man!

[@technithusiast]

Thanks im glad you enjoyed it!!

[@Smirk_Station]

For a first run... amazingly pull off. 👌

[@technithusiast]

🙏🏾

[@Smirk_Station]

@@technithusiast 🙏

[@RandallReviere]

One idea is to leverage the fact that ChatGTP can be its own best critic... so at the end of every 'action chain', have it ask itself 'is the last thing I produced relevant?' And if the answer is no, then do nothing. Having said that, I think you are really getting somewhere. This is a great project... keep it up and keep evolving it. Very interesting stuff.

[@n0izes]

"That's a great idea!" and an image appears of a bulb turning on. Seems like it made sense in some way to agree by turning on a light

[@cleverman383]

I'd be curious to know what GPT would say about situations such as turning on the wrong lights if you told it to explain its rationale in detail before giving any commands

[@Modus_Pwnin]

This series is awesome. I wonder if you use mm wave with esp bluetooth tracking to notify ke of your location in the house if it would do a better job? Although with multiple people in the house maybe you would need a biometric voice recognition to know exactly who was talking to ke

[@technithusiast]

I already experimented with that idea :) check out these two videos: - kzbin.infoxdL3j7x-NBM - kzbin.infoPmM3rIJ6zzE Ke is able to use the history data from the sensors to answer and complete location based commands

[@fr1sket363]

I wish everyone watched this video, it's a super balanced and unbiased view of AI autonomy. I think people forget there are sometimes more than one right answer to a question or even wrong answers. Your point of training from a large pool of essentially different view points both correct and incorrect ones, means exactly what you experienced, different outcomes to the same intent and or a different interpretation of your intent, that's unpredictable. I think no matter how good AI gets, it should never be unsupervised. This is just my personal opinion.

[@CrusadeVoyager]

Please create a series on deep diving on the Home Automation that you have created.

[@toleyk]

Pretty cool. I think you needed the guard rails on long enough, and memory for gpt to learn the correct way to interpret your intentions. Perhaps ask for clarification it confidence in the expected action wasn't high enough.

[@riflebird4842]

Wonderful experiment. I would suggest you to go to any one of the open source models and optimise it yourself for controlling a smart home. I think it would solve the problem. But like you said, even ML researchers don't know how GPT models are going to react. People did find out some weird behaviours, sometimes specific prompts unlocks whole new abilities, different behaviours and attitudes.

[@technithusiast]

The solution I have is pretty flexible as I’m able to swap out the LLMs. I’ll probably experiment with some different models in the future 👌🏾

[@riflebird4842]

@@technithusiast cool. Checkout huggingface for open source models. Waiting for your next update.

[@neezus3123]

9:53 had the same reaction😟This video was an amazing first impression, subscribing that’s for sure!

[@technithusiast]

How can anyone have any other reaction 😁 Thanks for the sub!

[@jjbankert]

Practical question: how do you parse the chatgpt responses? I sometimes run into the problem that chatgpt returns the expected response + some natural text. And then that needs to be filtered out or it will lead to parsing errors.

[@technithusiast]

Good question. I ran into a similar issue as well. Since I’m expecting the result to be in the form of a JSON object, I scan the response for the first open curly brace and delete everything before it. So far this approach has been working but it’s not perfect and would fail if GPT added additional text after the end of the expected JSON response. Lemme know if you need more details and I can provide you with code snippets.

[@seanaguilar2057]

"Ke, I'm scared" Ke: (processing. Horror is a popular form of human entertainment.) "You think you're scared now?" (starts laughing maniacally)

[@technithusiast]

😆😆

[@wild.legend-music]

A question popped in my head.. How do you expect it to know which lights to run on, if it doesn’t know where you are? (Note, I haven’t watched the other videos, but wanted to see what this is about) I’ve been thinking of something similar, like place 4 microphones in each corner of each room and just compare the which mics picks more sound when you say a command. That way it will know your exact location in the house and you don’t really need your phone all the time. It’s a bit of a privacy concern, but if you manage to make it secure so that no sound leaks out of your home server, you’ll be fine. Maybe use 2 servers, one connected to the internet for finding info and one offline that works with the mics and commands. Also you could use motion detection sensors, but the mics idea is better, because you can have multiple people in different rooms use commands and it will always know where the commands are coming from. I like the idea, but without it knowing your physical location in your house, I don’t think it can fully do what you really want. The microphones are the most ‘free’ and cheapest way to do it. Otherwise you can use cameras, which is a big nono, or wear some location tracking device, but then you have to wear it all the time. If you know a better way, I would like to hear about it. Thanks for the video, it was interesting and I will come back to watch the rest.

[@timothymaggenti717]

You are brave, I would not trust the tech that far just yet. But man it would be sweet if it was like in the movie's. Thanks

[@technithusiast]

I definitely understand people’s apprehension and they have good reason to be. With every technological push forward someone has to take those calculated risks.