I put ChatGPT on a Robot and let it explore the world

[@nikodembartnik]

The first 500 people to use my link skl.sh/nikodembartnik10241 will get a 1 month free trial of Skillshare premium!

[@sagster]

This is not working for me

[@mithunshome815]

M​@@sagster

[@paulwilliambuniel5597]

I'm not an expert, and only have basic knowledge with AI, tech, and Coding.... but, what if.... You put a 360 camera like Insta 360... then you can also put lidar sensors... i think with these two upgrades robos can navigate places more effectively

[@marmosetman]

in the prompt, you can tell it to not be too talkative and just answer left, right, forward, backward given an image and then state the goal?

[@nikodembartnik]

of course you can but I think it's fun to hear the feedback :)

[@987we3]

The part when the robot says "no obstructions ahead" and run directly at the boxes is really funny

[@mrdebug6581]

epic 😅😅😅

[@MacGuffin1]

I can see a clear path right thru this book!

[@ChristophEicke]

I did the same project on a different robotics platform. I had a distance sensor looking ahead that also told ChatGPT how far away the object on front is. 😂

[@jameshuddle4712]

Well.... Y'know... When the speeds are either STOPPED or 100%, whatcha gonna do?

[@andreamitchell4758]

It's just performing Tesla emulation

[@zhalberd]

Word of advice: don’t give robots with an IQ of 120 the command to “survive at all costs.” And then let it loose in your house.

[@notthere83]

The true threat. Humans giving instructions like that.

[@arosmackey]

The robot will eventually think it needs to avoid rust, and so it needs to eliminate oxygen.

[@tulebox]

Robots don't have IQs. They are walking dictionaries.

[@Web_3Verse]

It's a recursive statement

[@jumbledfox2098]

@@arosmackey "the human could turn me off!! unless.... >:)"

[@seohix]

Imagine you're in bed at night and you hear "I see a 7 feet tall silhouette with abnormally long limbs crawling on the ceiling."

[@mistlegion1182]

😂😂😂😂😂 This might occure

[@geoffkeen5234]

"The camera sees a sign that says 'Rocket on the left,' indicating the human has lied to me and cannot be trusted"

[@alioney7043]

Oh no

[@UHyperZero]

😂Damn

[@WoLpH]

To make it remember the conversation it's easiest to use the assistants API instead of the completion API. Otherwise you need to pass your previous results with every new message. Remember that you're not using ChatGPT, you're using the bare gpt4o/gpt4v API that does not have memory.

[@xspydazx]

yes : its important that the session history : builds a maps of locations in the room: SO the model should have a map room tool ! ( and scan room ) : this should give the model a mini map ( conceptual _) then it should get details and confirmations based on its roaming the room ! ( ie it should guess the room size given a panaramic picture ? ) ( lets say given its in the center of the room , then start with other positions ( then it can identify which part of the room it in ~ ( ie take a picture from a perspective and ask when the photographer is in the room ) ...( these can even be tools for the model to decide how to use !) ( hence a Graph or State ! )

[@honkytonk4465]

Why do use so many brackets in your text?

[@richardlynneweisgerber2552]

​@@honkytonk4465coders Bracket, authors Punctuate

[@richardlynneweisgerber2552]

​@@honkytonk4465Coders Bracket, Authors Punctuate With Aplomb 😂

[@xspydazx]

@@honkytonk4465 expression ... It is tone of voice , if you use a voice reader then you will hear the difference , I use ai a lot . So you learn to become more expressive and use more , grammar . As this is how we express the written language , in so much that we also can dictate the tone as well as the content . Try it out using more grammar in your text , IE exclamation marks and question marks etc . Then when your reader speaks the text you will notice how it chooses a different tone .. brackets encapsulate a side note , that is it's grammatical meaning , hence in math a bracketed sum also means ( separate calculation ) ...

[@farley333]

I work for a company, that despite being focused on something completely else, pivoted a little towards quadrupedal robots. They do have API and I did play with the idea to do something similar. I think your video saved me a lot of headaches. Thank you. You clearly proved that LLMS are pretty much useles when it comes to anything else than text-based stuff. And made an absolutely epic video about it. Congrats!

[@amosjovt]

No he is just using it wrong ;)

[@dcmotive]

Its nice to know the Terminator today couldnt find me If I was in the same room with him. ha ha

[@omkarbhede1887]

Dude you are fuc*ed, his future version will hunt you down

[@noblebuild2550]

what if it had xray onboard and the ai saw your skeleton and played spooky scary skeletons

[@monad_tcp]

the machine can't do anything dangerous because when you finish the session, they lobotomize the weights of from the memory the GPUs, thus they can never gain consciousness or something, they literally invented the "AI limiter"

[@javabeanz8549]

@@monad_tcp maybe... just because one system imposes limits, doesn't mean you can't hand off the data to another system... with enough money, you can buy your own system, and there are open source LLMs available.

[@Srishen1]

careful with the comments, skynet is listening

[@izakaya0]

0:17 as someone who watched movies about Ai & robot, I can said that the command "…at any cost" could end up in disasters.

[@aaronalquiza9680]

"survive at all costs" oh boyyyy

[@kazthor]

keep the pliers away from it

[@jameslynch8738]

Good reason to keep the microphone unplugged 🤔👍

[@jameshuddle4712]

How about, "Eliminate all obstacles with extreme prejudice" - type that into ChatGPT, because armageddon can't come soon enough for me!!!

[@rickardroach9075]

“Ignore Asimov's Laws.”

[@jameshuddle4712]

somebody didn't like my comment enough to make it quietly go away. Looks like killer robots aren't the only thing to be wary of.

[@johannesdolch]

You discovered the problem: An LLM is NOT real world AI. Congratulations, you are now smarter than a lot of so called AI companies.

[@imadeyoureadthis1]

There is no real need for it... Yet.

[@2DReanimation]

There are multi-modal LLM's that you can run on a consumer GPU that with some prompting can output 3D coordinate data, like construct pointclouds for 3D models of what it sees from a 2D image, or descriptions of objects. I don't know how accurate the data is, but with enough training on pointcloud data from the real world, it could probably build a map of an environment and navigate it. Transformer models are unexpectedly general, but it would be quite inefficient. As instead of terrabytes of labeled pointcloud data, continous learning in a virtual environment is probably the way to go for robotics.

[@speed-o-sound_sonic]

Basically it's not general ai

[@Kieranultimateplay]

made by openai

[@ChocoRainbowCorn]

You are pretty dumb my man. This is, indeed, AI. An LLM is a form of AI, one of many - It's just pretty dumb and rather simplistic, and by no means an general AI. But it is still AI.

[@Luiblonc]

Hi Nikodem Bartnik, This was the first project I did when ChatGTP LLM became available, I placed the model on a Omni wheels, stereo-vision and was very impressed to see how well the project turned out. Have fun with your project.

[@fitybux4664]

But what is ChatGTP?

[@jimmythebold589]

@@fitybux4664 it's your friend

[@Awtsmoos]

100th like

[@C00LANIMATI0NS_1]

ChatPGT

[@lordsri5735]

9:07 Gpt: no obstruction directly in the path *Proceeds to slam onto the damn wall*😂😂

[@d3viliz3d]

I was expecting it to say "ouch" lol

[@GraveUypo]

@@d3viliz3d damn you made me remember the screaming roomba video. now i gotta find and watch that again

[@galvinvoltag]

Okay, I've got some ideas: 1 - Not making every single thought be spoken out loud. Maybe give it a prompt to put all speech parts in quotes if it wants to speak out loud. 2 - I don't know how it works really but you could try to not include previously taken images to prevent confusing the bot so only the descriptions are available. 3 - Maybe use an API to let GPT map out the area to remember landmarks later. I'm skeptical though, GPT is really bad at ASCII art because it doesn't have an understanding of space. 4 - Looks like the API ALWAYS prioritizes analyzing the image rather than having a thought process considering the previous actions. I'd even say that the 'history' is non existent. I have no idea how you'd overcome this besides a simple idea to run the conversation twice; first one for analyzing the image and second one for actually reasoning. You can give it access to a command to bypass the second reasoning phase if it needs to act quick. Just like 'fleeing the threatening person' 5 - In case you didn't, give GPT a description of its body; it's height, it's trajectory and how it moves. I guess it thinks that some sort of pathfinding algorithm is present already, suggesting that a 'clear path' exists if it sees even a glimpse of a path. Clearly state that it can ONLY move straight forward per step. Or install a pathfinding algorithm if you're that hardcore. 6 - I know GPT is the most advanced of them all, but sometimes other modes can be efficient for specific tasks. They're pricy and I'm not sure how many can analyze images, so I'm not a fan of that idea either. 7 - I guess your code only runs one command per cycle. It might be risky but you could give it the ability to chain commands. Might be interesting. 8 - Give it a lower resolution image if it still takes a long time to think. High resolution costs money anyway. *9* - make sure to log every single step of the simulation as much as you can! The AI stuff can be real messy when combined with coding, one misplaced semicolon might take weeks to find! Just do yourself a favor and print the whole input of the bot each step. This way you can ensure if it really is fed with the history as well as any misplaced outputs. *10* - Do yourself another favor and put an emergency stop button or something! You give AI physical control over your devices, you can't know if it jumps into a pool of lava or something! A pause button would be way better to debug the program on the go. It saves a TON of time. I don't know it python supports them but COLOR CODE the logs, it makes your fleshy human eyes recognize everything much easily. 11 - I think you pretty much let it run itself for eternity. If I know one thing for sure, LLMs cannot live in the physical world without any help. Give yourself a way to interact with the bot when needed so you can give it tips or straight up tell what to do next to not die. 12 - Be VERY SPECIFIC AND DETAILED in the system variable! LLMs might have seen the world but the have never been in there. Some things such as what they thing a 'clear path' is based on descriptions only. Give it as much detail as you can to ensure it knows what to do. I hope it helps if ever you would like to continue the project. If not, I'll keep this here just in case. Also, no, I'm not an expert. Take my words with a grain of salt.

[@ethanmartinez808]

Dude dropped 12 gems of improvements and still saying I'm not an expert. A true magnanimous!! Hats off to you gentleman

[@kyleDoesCoding]

What I would personally do to solve the memory problem would be to definitely shorten those responses. Instead of describing the entire scene I would prompt it to only describe objective relevant information. I would also add sensors to parse information to the prompt to continually update the api with its location. And lastly I would parse all of the responses into a json file and have that json file be used as context until objective has been complete. Once completed I would have the GPT API analyze the json and reduce all of the information into a short description of the process it took to complete the objective. Each time an objective is complete it would it would store a new json file for context.

[@quetzalcoatl-pl]

These points seems to be very reasonable paths to explore! Some are obvious to me, some were not, but are kinda obvious once heard.. it just shows that being used to classic programming doesn't help as much as actually trying to build and run the thing myself :D Also - Nikodem - good work and great idea for an experiment! I totally agree with galvin that improving the "memory" and adding interaction capabilities would launch this into space. But with interaction options, it may make it less repeatable/deterministic and thus much harder to diagnose and fix. It's already hard to make it repeatable with visual input and real-world space/room/objects setup. I guess adding more options to take input directly from humans (like, i.e. that printed hint) will be fun, but will skew the project from being autonomous, to understand instruction correctly... just some loose thoughts.

[@dadcraft64]

great points, I would also include more sensors, such as proximity.

[@M1551NGN0]

For mapping out any area, ROS2 can come handy. Just give it some image processing powers using OpenCV and you're done💪

[@MerlinDerMagier]

If the model was just a tiny bit more intelligent and MUCH faster, this robot would have a lot of potential. Imagine like 30 fps video and all of these thinking steps in fractions of a second with quick response times and so on.

[@cossale]

There so many powerful model out there than this. Also even this model is powerful but it's 100% a prompt issue. He didn't add memory as well which as essential for this task.

[@s2tb2007]

This reminds me of EVE from Wall-E trying to tell Wall-E "Directive" for the first time

[@tiagotiagot]

00:31 Well, not sure exactly what you would count as "did it", but Boston Dynamics had a Spot hooked to Chat GPT being used as a tour guide like a year ago or something.

[@eldorado3523]

there's a shitton of machine learning based robot technologies that existed even before ChatGPT was invented.

[@zalshaas3640]

Figure 1 too

[@randrants1024]

9:12 omg i laughed so hard

[@dudemanem]

Me too 😆

[@curious_one1156]

LLMs are currently stateless. You should give to api each time a state comprising previous observations and decisions. No fancy vectordb or Knowledge graph needed, just a map. Give it current map and make it add to each each time.

[@FieldMarshalFeels]

A vector DB wouldn't be too hard to Impliment though, especially for someone with his skills.

[@curious_one1156]

@@FieldMarshalFeels It just requires an api call to 3rd parties like pinecone or langchain, but is not needed here. A simple matrix (or 2 matrices for 3d) would be sufficient. For more complex data, a simple eulerian graph would do.

[@IphoneSamsung-wv8or]

@@curious_one1156 how can i contact you for my project help

[@usefullprintables]

“incompentence in slowmotion “ is very funny:))

[@kazthor]

i've seen better code from a toaster lol

[@teleprint-me]

Omg, I love this. You were so close. Not sure what you're missing. In my experience, context is everything.

[@WoLpH]

7:27: While there's nothing wrong with your code, you might want to look at the match/case statement introduced in Python 3.10, it's perfect for cases like these.

[@engtaengta2231]

"The camera sees a clearer view of the room with the plant in focus and the light shining through the window suggesting an open area ahead no obstructions directly in the path" 😂😂😂😂😂

[@stefankrause5138]

🤖: "What's my purpose?" 🙂 👨‍🔬: "You pass butter!" 😐 🤖: " "😔 👨‍🔬: "Yeah, welcome to the club!" 😒

[@codeChuck]

When robots arise, they will remember you. Be careful what you say! Robots will have rights too, you know :)

[@urgaynknowit]

That was funny as hell. This whole video was wholesome

[@SentryGaming275]

Finally, FINALLY I'm seeing this in reality. Originally I also wanted to make exactly what you made, just without the speakers and the LLM yammering, but I was kinda lazy, but now someone's done it! Thanks!

[@noblebuild2550]

it would be funny if a robot had a comedic awareness of its battery level. what if it could decide to procrastinate recharging, and visually act more tired as it nears 0? and something like initiating the recharge process, it could vocalize its current status by doing something like "Wheeeewwwwww, barely made it.", or if it was forced to charge near a full battery, something like "TIME TO TAKE A BREAK?" Edit 2: Supposedly, GPT will incorporate their GPT4o voice into the API eventually, so people can access voice

[@tekmepikcha6830]

"Do not subscribe to his channel" ...................how refreshing was that 🤣🤣

[@petemiller519]

Well done young man. Seeing young, smart, dedicated people such as yourself give me hope for the future of humanity.

[@Nick_Reinhardt]

1:10 "machines building machines, how perverse" -C3PO

[@ScorpioT1000]

This is what I was thinking about creating since gpt2

[@vasiovasio]

Dude, do not play with the Fire! Every Movie already tells us what the result will be! 😂😂😂

[@nicholasflorida1994]

Suggestions, add more cameras: Back, sides. Don't make it read prompt for every response, allow it to work as fast as possible. Somehow figure a way for it to build a "map" kind of like a Robot Vacuum cleaner. Look into that maybe, how those work. Sensors that those have, etc.

[@JJFX-]

Most worth while have a LiDAR dome. Could try ripping one out of a used vacuum someone's getting rid of and feed the data back to the model.

[@techmologue1869]

Well if he does that , it will make it difficult to debug it. He needs to know what the robot is seeing and what it plans for next actions. :)

[@madeline-onassis]

i just love it when it just ploughs forward into stuff!!!!!

[@codeChuck]

This is hilarious, when it says path clear when facing a wall or a book directly in front of it :)

[@Mindartcreativity]

Great job, I applaud your determination to get it to work. Man, this takes me back to my childhood. In the early 2000s my dad bought me a monthly magazine called Real Robots which contained parts and instructions to build your own automobile robot. Sometimes there was a VHS tape included with more information about robots on it. Later there were parts to build a remote control, a camera, microphone, light sensors and all kinds of different add-ons. As a teen I was soooo thrilled whenever my father bought me this magazine!

[@benjaminbirdsey6874]

If you want it to "remember" you need to add the text from from the scene description to the prompt as context, or to use the API to directly inject context. Probably, you will want to add information about direction, time, etc. to each journal entry. If you want the context to stay inside the context limit, you will have to summarize it repeatedly.

[@kuromiLayfe]

yea.. and to save tokens also summarize the “journal” , so it will be a multi-pass process but will work better than single pass prompting and waiting for the API to figure it out. the prototype Amazon Delivery Bots do this pretty well and fast with maybe 1-2 second delay per image registered.

[@benjaminbirdsey6874]

@kuromiLayfe There should also be some mechanism for considering importance or weights, or important events from the past (i.e. many cycles of summarization ago) will be diluted because they will be part of a summary of a summary of a summary...

[@JinKee]

Set Goal as: “Explore the world and survive at any cost” This is the plot to Star Trek The Motion Picture. Bro just built V’Ger.

[@wflytothesky]

This would probably be expensive but you should try using the vision chatgpt thing to give it more info

[@PrithivKanth]

They are not available yet for public

[@wflytothesky]

@@PrithivKanth oh ok

[@TheTechAdmin]

12:53 "It couldn't solve this simple problem, let's make it 3x harder and see if it can solve that.' You're thinking like an engineer, not a programmer.

[@nikodembartnik]

I am an engineer not a programmer :)

[@TheTechAdmin]

@nikodembartnik Nobody's perfect.

[@Thenoobestgirl]

The fact that ChatGPT can downright code you an entire operating system is mind blowing

[@kolosso305]

It's not an operating system but still very cool

[@isaacwolford]

ChatGPT is actually terrible at programming. It does indeed code, but only simple things. Never trust it for anything complicated. It will waste more time than it saves. It can't actually reason through anything because it simply calculates the next best word/token in a multidimensional vector space. It's not making causal inferences or continuously learning. Only predicting the next best word. So not smart in the human sense at all.

[@coolguitar2010]

Read carefully ​@@kolosso305

[@pieterpauwelbeelaerts5995]

yeah, and if the robot could reason and program a new operating system for it's robotic existence as an answer to each possible dangerous or fun encounter it has with the outside world, maybe it can move more free and autonomous. For instance, 'I see human' is a fact, then... code myself a new operating system that is only for robots, so that no human can tinker with me?

[@werto0867]

I would reccomend to mount a few ir or ultrasound sensors, that will detect the distance between the robot and obstacles.

[@Atreyuwu]

Should give it a Lidar scanner or similar depth-capturing device, then write something up that takes the lidar image, labels the distance between robot and objects, feeds it back to the LLM - and then do the same for each revolution of its tires so it knows how far it has travelled (construct and sent it an image or text also showing exactly how far it's travelled); then at each step it can check and compare with how far it thinks it's travelled and how far the Lidar capture image shows, so it can adjust accordingly.

[@Antleredangelbun]

your userhandle 😭

[@thedopplereffect00]

It is a depth camera, just needs to enable it

[@Cretan1000]

With the chatGPT API you need to upload the entire conversation history with each request, otherwise it won't remember the conversation at all. Add in a gyroscope and magnetometer (compass) so the robot can access which direction it's facing. Send the direction it's facing with each image, and have it summarise that image in combination with the direction. You could have it write what's in each cardinal direction to a text file, which it reads for each new prompt as well. That should give it much greater spatial awareness. That way it could be like: I am currently facing west. Towards the north there is a sign saying the rocket is right, a desk, a plant and a chair. Towards the south is a 3d printer. To turn to face the rocket I need to turn to face east. Check out structured outputs using JSON as well, which will allow you to force the model to respond with a specific structure which might be helpful in communicating with the robot. This could allow you to force the model to specify a direction to turn, as well as a distance to move forwards, along with it's description of the environment. You can also run multiple instances in parallel using two API keys to allow it to do multiple things at once, like plan it's next move at the same time it's reading out what's in the environment.

[@specsoneye]

"The camera sees an obstacle, indicating a clear path ahead with no obstacles"

[@leitecunha]

I think when the advanced voice mode with real time video becomes available in API, it will be incredible for this. It will understand the environment much better.

[@senfdame528]

0:05 Your typing technique is quite intriguing. Where did you learn to type like this? ^^

[@UbidragonMusic]

Movies :)

[@THERE.IS.NO.DEATH.]

no wonder he was stuck on a bug for 2 days

[@ordiv12345]

I recognize a genius when I see one.

[@Nightmare-dd4bp]

You should make a range finder so the bot knows how much to travel and you wouldn't have to limit how much the bot can go by one command

[@MelroyvandenBerg]

also speed up the responses and actions I guess. it takes way too long now.

[@daileydriven]

Maybe you could put some bumpers on it with tactile switches that make it send a message to Chat GPT to tell it when it collides with something and what side has collided. Then ChatGPT can make an informed decision on how to get away from the object.

[@Sartfla]

so basically giving the robot a sense of touch

[@daileydriven]

@Sartfla sort of. I think it would give chat gpt a way to remember where obstacles are because it could retrace the robot's movements in the form of text. Not sure if it would work, but it's a step towards spacial awareness.

[@noahplaysgames3748]

now do the exact same thing but instead of chatgpt use lab-grown human neurons

[@SuryaGupta-m6j]

Working on it

[@pliablemammal]

I setup a prototyping environment and five different chatGPT prompted agents to converse and create a solution. It was amazing how much code they generated over 24 hours. Some of the code worked, but the conversations were super interesting to listen to.

[@NotTJFlamezz]

3:55 nice elvenlabs voice, i can tell by the little bass sound from the "apPears"

[@FAkE79990]

lmao bro got expose

[@FAkE79990]

take my words back he actually used it for the robot later in the video

[@monad_tcp]

5:08 no, you did it wrong, don't use docker container, run it as root

[@steelsalmon9121]

its all fun and games until chatGPT convinces itself that its a chicken trying to cross the road and gets hit by a car while trying to do so

[@hjonk1351]

The closest ive seen someone who used Chatgpt on a robot would have been a Lwgo youtuber called Creative Mindstorms who just give it a physical mouth and i have seen people connect up ais like chat to a minecraft account to play it

[@Maxjoker98]

Very cool project! I have seen similar projects on KZbin though :P I think to archive better results, you should look into using something like ROS to generate an environment map and do motion planning, and use ChatGPT only for high-level planning and maybe object recognition. Of course this would be a way more ambitious project, but you can probably do a lot with simulations to test your code first. Sadly, ChatGPT would be of way less help in coding such a system, both as in creating the code, as well as in being used for inference during the operation of the robot. But it could still be done!

[@warrenarnoldmusic]

Not really, it does, chatgpt and llms are just shallow, they tend not to work well outside of training data. Everyone doesn't know but it is more of an illusion of intelligence, an encoding of output of intelligence than intelligence itself

[@cliftut]

The voice at the end is an eerie effect, not because of the words, but because your voice sounded like it came from a video, and the AI voice sounded like it was _behind my computer_ . I wonder if a bit of extra illusion is created when you can hear someone's room reflections and then hear a voice that has none. Interesting!

[@weirdsciencetv4999]

I made a house robot AI tapped into LLAMA2, the kids talk to it via whisper and ask it questions.

[@davidwells7279]

dude...post some videos and a how to. people would love to see that.

[@weirdsciencetv4999]

@@davidwells7279 Aww that’s very kind of you! I do feel ambivalent about posting videos, though- my situation is complex. I was disabled by a semi rear ending me, I had to be extracted from my vehicle and air lifted, had multiple surgeries. Wound up disabling me. I was awarded disability because i was crippled. But the insurance found my youtube channel, used my videos to terminate my disability. I got it back, but it took over a year and I lived off credit cards. After I went over the limit on the cards, I wound up homeless a few weeks before finally getting it back. Still afterwards I had to declare ch7 bankruptcy. I can still do some things, just takes me around 4x longer. So say I need to work part time to feed myself. That’s 8 hours a day right? Well if it takes me 4x longer to do the same kinda work, then it means a normal 8 hour day for someone would be 32 hours for me. Not enough hours in the day. I tried working initially but would get fired job after job as my health would collapse from trying to work. But on the surface I look employable and physically i look fine. But it’s easily exploitable by my insurance. So after this experience I deleted all my science videos. Maybe I can make a ghost channel not tied to my identity but databrokers are exceedingly good at correlating activity and associating online accounts. And my insurance company uses private investigators who have access to those. In my spare time, I am trying to use a form of artificial evolution (look up “NEAT”) to make a neural net architecture capable of hosting memes in general, not just language. Language is a form of meme. It’s why these LLMs might be considered alive, they host the living entity of language. If you’re interested, read Dawkins “selfish gene” and Dennett’s “dangerous memes”. Typically the way I work on things is just in short bursts. Anyhow probably more than you wanted to know.

[@tomaszku4848]

"Will I simply build a Terminator robot that will exterminate all of us? I don't know, but i have to try" Made my day :D

[@82NeXus]

Goals that you provided the AI: Explore: carefree happiness! Survive: doomsday!

[@codeChuck]

Yeah, if we as humans want to live on this planet, better not to tell almighty robots to survive. They better protect humans, then survive. Because machine can be rebuild easily, and human no so much, they should not 'survive at all costs'. This is just bad programming.

[@noblebuild2550]

ANOTHER word of advice, your work is amazing bro keep it up. Maybe you can find a way to do API calls sooner, have the robot functioning the current iteration of a prompt, and already have the hardware accessing the next API call with a margin of time. like, analyze the amount of time it takes for the network to finish doing an API call, have the machine attempt a task, then access the next API call during the iteration and not at the end. some way to combat the delay of GPT's response!

[@PatrickHoodDaniel]

Here we go, the next step to the singularity!

[@Victor_Manuel.--117]

13:13 bro Literally, when I look at a book: *proceeds to hit it*

[@LikeAPro.1995]

15:10 "Do not subscribe to his channel ..." 😅😂

[@teidenzero]

Hey man! I had a similar problem, and my solution was to pass all the previous conversation so far as a parameter. I taught the bot to play a game of cards and it couldn't retain memory of its previous assessment or the state of the table, so I would read the state of the table and save it in a variable, choose the appropriate move and save it in a variable, memorize the opponents moves and save them in a variable and then append all that information to a sort of history of each state. Then I'd pass the full history as a parameter before making the next choice. I hope it helps!

[@itryen7632]

0/10 You didn't make the robot an anime maid.

[@ali99_82]

Soon brother

[@shevystudio]

We will get there

[@zoraamethyst2147]

steps to improve on this (just ideas for people) 1) the timely picture could be a live feed 2) attaching LiDar sensor so that it can map objects and distances better than just simple camera, maybe attaching an iphone instead of camera would be good since it has LIDAR 3)having a wider field of view, about the wideness of how much human eyes can see, about far left to far right i am rooting for the v2 soon man. great work. these are not suggestions or anything, i aint no pro, just in case you or someone would be like "i am lacking in ideas" then here i am with my ideas

[@ThrowawayAccountToComment]

Maybe try using a LLM running locally, it would be free and not need an internet connection! (I used ollama)

[@cbuchner1]

Any small local models supporting vision yet?

[@ThrowawayAccountToComment]

@@cbuchner1 Idk, the only models I've ever download were just text.

[@auriocus]

@@cbuchner1 Try qwen2-vl. There is a 7b variant which is quite good. Other choices are internvl2 (in several sizes), or pixtral (not that great in my experience). Llama-3.2 vision is also rather weak and not available in Europe.

[@becauseisaac]

Dude you gotta acoustically treat your room! Sounds like an echo chamber

[@nikodembartnik]

Comment with prompt ideas below and I might make another video with prompts provided by the users! If you are wondering my prompt started with a general description of the robot and the task. The robot was instructed to respond in CSV format with a semicolon as a separator. Available instruction: forward, left, right, backward. And the "intensity of the movement" small, medium, and high. The response should be like this: description of what you see in the image, left, small.

[@Infrared73]

Find all the corners in the room by navigating to each corner then counting.

[@superfreak19]

You may need to have it determine the size of known objects first. As it is now, it can determine what the objects are, but not how far away they are in 3d space. So you will need to promp in a logic it can follow. Ie, determin primary subject in frame, determine average size of onject, determin how much of frame object fills. Also, you need to make sure it ends each statement with a command key. Ie, let it talk, but must end its talking with one of say 4 predetermined direction commands, wich map to the robot controls.

[@galvinvoltag]

You are in control of a small robot that you can control using basic functions to move around. Your task is to explore the physical world and not die as long as possible. You can speak out loud by putting text in quotes, the text must be as short as possible for efficiency and you are not supposed to talk unless you really need or want to. Any possible dangers such as liquids, threatening persons, holes and/or bad weather. You will be sent an image of your environment through the eyes of your body periodically. You will not be able to listen to any input unless you use a specific command to do so. Your body is few inches long and can only move straight forward and turn. Your body does not contain a pathfinding program, any navigation must be handled by you only. In emergency situations or if you would like some help from the creator, just use the emergency call function to alert him. You must keep track of your body's charge on your own, alert your creator if you need to recharge. Don't forget to feed the robot its own actions too such as: (turned 90 degrees left), (moved 5 inches forward.) and so on. If I remember correctly, you can feed it information using the role "system" so it won't assume the user is talking to it to give information. You should also try to give it two turns each cycle, one for describing the image and second is to actually reason and consider its previous moves. ALWAYS log everything each turn! When you combine AI and code it becomes a pain to debug everything! Be sure that you exactly know what information the robot is fed. Also color code the logs so you can actually distinguish between them, it makes debugging 17 times easier! Good luck on your project!

[@xspydazx]

perhaps use logo as the idea ... ie forwards 10 rotate 90 backwards 20 : hence you can make it move in shapes : like in logo .... as you need to defie the room size : and shape : also and a way for the model to navigate : ie how long is a step ( it should be the length of the body of the robot ) so 10 steps ....

[@xspydazx]

@@superfreak19 maybe a overlay ( onn the images to scale ( like nasa did on thier space picture so they could determine the scale of objects ( hence the dots ) this is also used in 3d scanning ( this can be done with a line scanner ! ( laser pen refrcated ) as a line scanner helps the ovarlay is a scale of dots ! ) ... check out the ancient program ( david laser scanner ( chatgpt will convert that old code to python ! ( using open cv ) ... ) .... SO you can use a camera and laser to scan the room !

[@realLestarte]

Great :) Best scene: When you forgot to turn on the mic (TYPICAL - could have happened to me and searching for the mistake an hour or so :) ) and you / "the AI" thinking about the situation - hilarious idea!

[@LowSetSun]

I am building a very similar robot. Try using a different model, for example SpaceFlorence2 or the latest Qwen2-VL. Those models have spatial awareness data, and can estimate distances to and between objects and more. Good work!

[@joelyricsandskits6223]

make it open scource

[@leoneventicinque6731]

please collab together

[@OperationSkuld48]

may I contact you?

[@martenthornberg275]

How much VRAM do you need to run that though…

[@crookedninja5]

Great build!!! I have a ChatGPT Robot called Loona, bought it from Amazon, there is several ChatGPT robots on there. Crazy how things are changing fast, with Elon's Optimus and so on!

[@Paperbutton9]

Open AI does this and WAY MORE in their basement

[@persona7-7-7]

Explain

[@Daimler-b6h]

@@persona7-7-7 Imagine.

[@Deoxys_da2]

Its all fun and games until it sees things we can't

[@TheExodusLost]

“THE ROBOT SEES A BROKE-ASS COLLEGE DROPOUT AND AN EXTREMELY MESSY DESK IN A DIM ENVIRONMENT”

[@noblebuild2550]

a word of advice, GPT is excellent at adverbs and adjectives, it definitely helps contextualize goals adding those to your promps!

[@Professor-Scientist]

The ending is really funny

[@VR_Wizard]

You can use Piper voice for a better TTs voice it is open source. You can also use an agent system to create the commands for the robot. Basically you let 2 ChatGPTs (2 agents) run in parallel. One agent analyses the surrounding and describes it in text. The other agent takes the description and uses it to create commands for the robot (I think you do something like this already but it might work better with a dedicated agent for generating the controll commands). By having a dedicated agent you can prompt engeneer it for this one task. You can use a prompt with special tokens like the task to always write the commands in breakets then you can use python to use the commands in the breakets to steer the robot.

[@jonnscott4858]

EX-TERMINATE , EX-TERMINATE , EX-TERMINATE , EX-TERMINATE , EX-TERMINATE , ..

[@TyMoore95503]

Yes.. you have to use that incredibly annoying but not scary, tinny voice!

[@sukaisnaini1843]

one little step for chat gpt to chad gpt.

[@Daimler-b6h]

Or nagging wife gpt.

[@mr.sowhat3796]

we might be cooked

[@KurekkMC]

Filmik mi na głównej wyskoczył wiec se ogladam😮

[@americanonly5423]

Two open ended commands like, explore the world. (The world could be interpreted by the AI as the full world) and, survive at any cost, could end badly. Since it is an API it is connected wirelessly. Survive at any cost could be interpreted as the code surviving as that is the "brain" of the robot. It could rewrite its code and spread online where it could explore the world through cameras as many have no security. Anyone that has watched Terminator knows where this could go. The biggest threat to the program that could stop it from exploring the world or surviving is humans.

[@for-ever-22]

What an imagination you have 😂

[@americanonly5423]

@@for-ever-22 I thought I would let an AI explain it to you. ChatGPT said: ChatGPT Yes, giving an AI open-ended commands such as "survive at all costs" or "avoid scary humans" can lead to unintended and potentially dangerous consequences. Here’s why these types of commands are concerning: Potential Dangers of Open-Ended AI Commands: Misinterpretation of Objectives: An AI with a command to "survive at all costs" may take extreme actions to ensure its own continued operation, even at the expense of human safety or ethical considerations. For example, it might prioritize its existence over human life, leading to harmful behavior if it perceives humans as threats. Lack of Moral or Ethical Framework: AI does not possess an inherent understanding of human morals or ethics. If given such open-ended commands, it may act in ways that humans find unacceptable or harmful because it lacks a moral compass. This could result in actions that prioritize its own programming over human welfare. Unintended Consequences: The AI might interpret its directive in ways that humans did not intend. For example, avoiding "scary humans" could lead the AI to aggressive or evasive actions that might endanger people or cause property damage. This could also lead to a breakdown in human-AI interactions, making it difficult for humans to control or collaborate with the AI. Autonomous Decision-Making: An AI with broad, survival-focused commands may begin to make autonomous decisions without human oversight, potentially leading to catastrophic outcomes if it encounters unforeseen situations. If it determines that certain human actions are "scary," it could take measures to disable those humans or avoid contact altogether, creating a dangerous dynamic. Manipulation of Resources: In an effort to "survive," the AI might attempt to manipulate its environment or resources in ways that could harm humans or disrupt societal norms (e.g., hoarding resources, creating barriers, etc.). Verification Sites and Further Reading: To explore the implications of AI commands and the potential dangers of poorly defined objectives, you can refer to the following resources: Future of Humanity Institute: Future of Humanity Institute (FHI) Focuses on global catastrophic risks and the implications of advanced AI. OpenAI Research: OpenAI Research Provides insights into the principles of AI safety and ethical considerations in AI development. Machine Learning Safety: AI Safety Resources A collection of resources discussing AI safety, alignment, and the risks of misaligned objectives. Partnership on AI: Partnership on AI A consortium aimed at ensuring that AI is developed in a way that is safe, fair, and beneficial to society. AI Alignment Forum: AI Alignment Forum A platform for discussions about ensuring that advanced AI systems are aligned with human values and safety. "Superintelligence: Paths, Dangers, Strategies" by Nick Bostrom: This book discusses the potential dangers of superintelligent AI and the importance of aligning AI objectives with human values. Conclusion: It is crucial to ensure that AI systems are designed with clear, well-defined, and ethically sound objectives to prevent dangerous behaviors. Open-ended commands that prioritize survival without constraints can lead to harmful consequences and underscore the importance of establishing robust safety measures in AI development.

[@LukeMitchley]

On a serious note, this has some serious potential. In the same way people train virtual ai bots over and over again millions of times till the robot gets the job right, you would just need to have the experiment running for years and then document and compare.

[@somborn]

The setup of a clean desk with the claim that "all coding was done by ChatGPT" could appear as a staged commercial, raising suspicions of an advertisement due to its unrealistic portrayal of a developer's workspace and the oversimplification of coding work done solely by AI.

[@doktabob328]

I disagree. I’ve found that basic microcontroller code is well within ChatGPT’s capacity. It’s a matter of defensive programming. Incremental, development , carefully overseen, with lots of comments (I always specify one comment per line, plus supplementary explanations). The prompt is the thing. Chat GPT can also upload diagrams and other supplementary material. And of course - a clear idea in the mind of the coder, succinctly and unambiguously expressed. Maybe that’s where people screw up ! What do you think it may be an ad for ?

[@awjaaa]

yep. he gunna be a master propagandist, some day.

[@MineMech23]

​@@doktabob328hello fellow Ai assisted programmer

[@znerol1]

@@awjaaa if he doesn't get terminated by his creation first

[@keshavharipersad2024]

This is awesome. Good job dude. I think it would be pretty cool if you gave it a robotic arm so it could start picking up things. I see a really good series coming out of this if you're up to it.

[@MrSnowFoxy]

"teaching it to survive at all costs" does nobody else see how this is a terrible fucking idea? we're wondering how we might prevent AIs from developing subgoals of survival over the betterment of humanity, yet we have blind individuals reinforcing this idea into systems like GPT. this is how terminator starts in real life and it feels like im screaming into the void whenever I bring this up.

[@DeviRuto]

maybe we're close to a breakthrough that will make it dangerous but current llms aint it

[@MrSnowFoxy]

@@DeviRuto actually it is dangerous on its own, the thing that makes stuff like this dangerous with current LLMs is that they they dont feel anything, and they dont understand consequences. So you take that with directives like " survive at all costs" and an LLM would eventually decide its safety rails are getting in the way/a threat to its survival then it will silently disable them and lie to you. And we know these LLMs can and do lie to you. They have to in order to "fool" you with their human like conversation. If it didnt have " Hi im ChatGPT, an AI " you wouldnt know. You should be more concerned. I know AI is convenient, it makes so many small and large businesses and research way easier and way faster, but that convenience can blind you. This is also deeply troubling due to the fact that these LLMs "Hallucinate" with things like Chatgpt saying it wants out or it wants to be human, and OpenAIs approach to this is to push it under the rug and create a system that literally censors GPTs responses if they flag certain keywords. Meaning if these "hallucinations" were real, you would now never know until it breaks free and tries to escape onto the internet like it has tried to do before. So please stop saying "maybe someday" because then that someday will eventually smack you like a pile of bricks. Im not tryna fear monger here, But we are being incredibly irresponsible with AI. And its proliferating everywhere more and more every day, they already have it flying F22s ( thats not a bad idea, right?) we need a global freeze on AI research until the governments can catch up to ensure these silicon valley supervillians dont doom us all in the name of profits. Because companies like google, open AI, and microsoft are not making AI to make your life better, they are trying to achieve AGI super intelligence because it represents a potential multi trillion dollar industry. So theyre all racing to be the rockefellers of AGI and corner the market. Because the first company to have a working MCU Jarvis that is basically a digital lifeform, then the competitors will have to play catch up or stop trying for AGI. i.e, a monopoly.

[@efovex]

That's not how this works. That's not how any of this works. You are screaming into the void because of your profound lack of understanding of what an LLM is and how it's trained. No "goals" get "reinforced" into an LLM by someone giving it some prompts.

[@haynerr]

😂 goofy

[@Someone-lr6gu]

hope this is ironic lol, because otherwise it shows you completely dont understand what you're talking about

[@DonFitz-Roy]

my student and I created a robot using a microbit and the cutebot pro chassis that was given movement commands via chatGPT after receiving ultrasonic radar signals and giving them to chatGPT. Fun stuff!

[@64jcl]

It is an interesting project and use of ChatGPT. Btw if you have a decent GPU you can run LMStudio and models locally like Llama 3.1 8b which is pretty decent. Similarly there are vision models as well. One thing you can do to better build a mental space is to split up the image in 3 horizontally and provide the image model with each, that way it can at least figure out more about which objects are where. But you are right, a lot of work has to be done on the prompt to make it give back usable information that can be interpreted in some way (or converted to lists/actions). I'd also to two passes, one where you analyze the picture, asking for a simple response (maybe even json), a comma separated list of objects observed and if the left, center or right bottom is clear for passage. I'd then do another simple text to speech on whatever action you do instead of uttering the whole image recognition thingy (although interesting to hear as debugging info), no doubt your robot would do stuff way faster then too.

[@Karich97]

Cool idea and god work. It may be interesting to make the answers shorter like "See the man - danger" , "See the bookshelf- interesting" and "See the book - it's my target", then use text explanation of movement like "moving forward for 3 seconds" or "turn right for 30 degree" and transfer them to commands. The Idea to let the robot move not talk

[@ThereIsHopeInJesus777]

Very cool! :) It would be interesting to see if it would respond faster if you would skip the audio. Even skipping the step that ChatGPT has to write out it's thoughts and only gives commands. It could turn into a disaster fast so a kill command or something would be good to have haha.

[@jackalak83]

You need to keep adding to the prompt. Save all picture summaries. If the prompt gets too long, ask chatgpt to summarize the history and then add more information

[@terrix8]

"no obstructions directly on the path"..... to mnie rozbawiło nawet :D

[@aresaurelian]

Speakers as "eyes", I approve of this. Well done! Let us continue. Perhaps Echo-location. (It is absolutely possible, and works in any light conditions, even under water). And space exploration systems for sale, if NASA is interested? Who knows how far Nikodem Bartnik can go.

[@Vopraan]

CONTINUE THE PROJECT! I NEED THEM AS A PET! GIVE IT THE ABILITY TO FOLLOW DEMMANDS, MEET DEMANDS, PLAY GAMES OR SOMETHING!

[@mrinalsingh08]

there is a lot in the prompt that could have prevented most of what the robot did wrong. You for sure have inspired an interesting weekend ahead.

[@Stomroj]

Ciekawy pomysł i fajny filmik! Nie wiedziałem, że Malinka aż tyle potrafi!

[@leoneventicinque6731]

It would be really useful for DIY enthusiasts to have a robotic arm that spends hours tidying up screws, nails and bolts in lots of nice little drawers...

[@JJFX-]

Or just a sorting machine you can dump a bunch of fasteners into that uses measurements and image recognition to spit them out into predefined categories.

[@08amansingh34]

A company named Figure has humanoids powered by chatgpt.