This mini GPU runs LLM that controls this robot

[@nikodembartnik]

The first 500 people to use my link skl.sh/nikodembartnik12241 will get a 1 month free trial of Skillshare!

[@X862go]

Amazing work, mate 👏

[@akissot1402]

it wouldn't be better if you train your own vision model, that only recognize obstacles and rives the bot, and for more complicated decision making use an LLM

[@StevenIngram]

I hope you get the Jetson working. I like the idea of it being self contained. :)

[@stedocli6387]

At 10:08 You said "If you persist, sooner or later you will run out of problems to solve" this indeed is a great place be. Nice job!

[@MilanKarakas]

What is missing here is a memory. Llama can understand few things and may have small amount of memory. But, after you cycling power, it forgets. It will be great to write python script and record all conversation. Also, some type of 3D mapping, where robot can store past experience and mark the obstacles.

[@GoodBaleadaMusic]

Even just something that captures basics and last few moments context.

[@rodrigogomes6086]

Also I think it would better if he gave data from some sensors, like the distance between the the robot and the obstacles ahead

[@ezradlionel711]

Bro solved sentience with a sentence

[@peacekeepermoe]

@@rodrigogomes6086 Yes. Also a way to map the room, like robot vacuum cleaners do. It will help with navigation around the apartment instead of relying on the camera. The vacuum robots don't need camera or lights, they can work in the dark/night.

[@Larimuss]

You can do this with a database connected to ollama and store in the vector database and use ollama RAG or any RAG really.

[@domramsey]

I think the biggest issue here is your overall approach and your prompt. You have a distance sensor that gives a precise result in cm, yet the quantities you're using are arbitrary "low", "medium". If in your prompt, you tell the LLM the nearest object in front is (say) 85cm away, the nearest to the left is 10cm, and to the right is 200cm away, then ask it to output an angle to turn and a distance forward to travel. So it will come back with "Angle: 20, Forward: 50" or similar, which should be easy for the robot code to process. Make every move an angle followed by a distance, but use actual measurements. Your prompt could probably also do more to get the LLM to guess at the distance the objects it sees are likely to be from it. Oh, and get more distance sensors and mount them at 45 degrees left & right. I really feel like these should be the primary input for guiding movement. Yes, it's entirely possible that won't work at all. :)

[@SuperFetaCheese]

It cracks me up how it constantly accuses your place of being cluttered for no reason.

[@AngweenAnnora]

i already imagining a super intelligence roomba with their greatest mission, to clean the world.

[@M13RIX]

Man, your videos are so inspiring! They significantly help me not to give up on my own ai projects. I would love to see better improvements in this one, for example complete rejection of paid serviceces in exchange for local, but still high quality ones (for tts you can use coqui xtts - runs localy, has a realtime version + you can clone any voice)

[@GhostMoney0007]

great idea same thing I was thinking I would start learned a lot from you

[@Math2C]

Here's what you should do next. Use a computer Vision software to identify only the names of the items in the room. Let it draw bounding boxes around the identified items. Combine the area of the bounding box with the distance from your distance sensor to determine its size. Im not sure you did this already but your robot needs to know its actual location. Use the LLM to distinguish between objects that are permanently placed and those that are laid out. Record the various direction that the rover has looked already. So for each object the rover should know its size, relative direction and the distance it is away from it. Provide that information to the LLM finally to determine which direction it should move. Or if it should rotate.

[@OriNachum]

With Hailo-8L/Hailo-8 you can do that on Raspberry Pi at surprising processing power

[@soeasy22]

Imagine a robot equipped with 16 NVIDIA H100 GPUs, running the 405B parameter LLaMA model, packed with sensors, exploring the world.

[@GlobalScienceNetwork]

Yeah, this is a great thought. Easy to create and super powerful. You just need to give it a platform so it can interact with the world as well. We will see many products using this coming out soon. It could be similar to the Tesla Optimus humanoid robot with very little development.

[@DearNoobs]

i love this project, wish i wasnt so far behind on all my other projects becasue i want one of these too!! hahah GJ bud

[@catalinalupak]

Great progress on your project. I like your attitude and thinking. You should also try n8n for more local logic on that Jetson Orin Nano. Also will help you to build a map of the environment and have it stored locally, this will speed up also the decision making. Looking forward on your next steps

[@RemoteAccessGG]

I think you should make the robot remember previous image outputs (if you haven’t already), so it will have some logic. And also add a lidar sensor if you find that camera hard to setup. Giving the information to the LLM will be tough, because it cant understand what are a bunch of random numbers given to it.

[@SLRNT]

i think the llm could understand it if given the "format" of the lidar data. e.g an array of 1,2,3 and telling the llm first number(s) mean distance to left, 2 for distance to front and 3 meaning distance to right. ofc the array would be longer and you could average the numbers or just separate the directions with code

[@Davidfirefly]

that would require some machine learning implementation

[@pimf-youtube]

Great video! Have you thought about running one model only for navigation and a second one for talking? Maybe it will be smoother. Or add something like Lidar for navigation (like some vacuums). No idea how this stuff works but maybe you do. Definitely subscribing in hope for the next iteration.

[@v1ncend]

When I saw your first robot, it brought back memories.

[@ginogarcia8730]

oh my goshhh finallyyyyy, been trying to find something like this

[@MyPhone-qg2eh]

But your text to speech isn't local.

[@colinmcintyre1769]

You don't want it to be.

[@slapcitykustomz1658]

@@colinmcintyre1769 Why not? Nvidia has local (llamaspeak) text to speech and (Open Ai Whisper) For speech recognition both library both can be ran locally on the jetson

[@ChigosGames]

@@colinmcintyre1769 why not? If the computer can create beautiful voices as well then everything could be locally local.

[@colinmcintyre1769]

@ChigosGames you want to utilize as much compute as you can for the best results, I'd asume. By trying to do everything locally, it's instantly much more expensive and less practical.

[@ChigosGames]

​@colinmcintyre1769 I fully understand you. But to outsource everything to paid API's, real life products will be unaffordable. Imagine making a product that only consumes API's, you could only sell it for a hefty price with steep subscription to it.

[@yt742k]

we are now in age of "talking animals" this prediction is so spot on said

[@erutku]

alas a great experiment(s). well done.

[@andyburnett4918]

I love watching your videos. They are very inspiring.

[@Gabokor-76]

Bro is creating the next terminator

[@murloc6634]

excellente, thanks a lot it's a perfect example. for distance you can try ToF RPi Camera, I didn't tester yet

[@anonym_user_nksnskdnkdnksndkn]

Do you think you could make a drone, controlled by GPT? would be sick XD.

[@manuel_elor]

Up

[@sergemarlon]

Seems possible. Drones can hover in space, acting like these robots without moving their wheels. The issue I see is that you would need a large drone in order to handle the payload of the electronics. It may be possible to stream the video from a drone to a stationary PC which then computes and sends the radio signals to the drone.

[@sKrible144]

you should give the robot a glados voice from portal

For very simple things, a microcontroller could be nice to learn programming, but IMO i think something like raspberry pi (computer with gpio) is much more useful for robotics to start with. Imagine you are creating robot, and you want to change code, see what the code is doing, see camera etc... you find bug in code, you just ssh over wifi, change code with nano, run code, see what it does, etc. Now i agine doing it with microcontroller. Any bug, you need to get to the robot, turn it off, plug usb cable, program it (in case for arduino wait for compile...), unplug, power.... it gets tiring pretty quick.

[@LukaNegoita]

Have you considered adding additional sensors that get fed to the LLM? For example, a distance sensor and the prompt that goes to the LLM is the image description plus something like “and the distance is xx centimeters in front of you”

[@OmPrakash-ai]

I feel like adding more sensors, like LiDAR, could help the LLM make better decisions. Also, what if all the data from those sensors and cameras were used not just for reacting, but for planning ahead and executing smoothly? It might make the robot feel less… stuttery, you know?

[@adolfoquevedo7429]

thanks, great video!

[@markusstaden]

I like the idea of using the vision capability of llms. You could try to preprocess the image, for example using depth estimation or using a lidar and putting the data on an image. Smaller Models might have problems understanding the data, but that could be solved using fine tuning I guess.

[@Confuzer]

I build a chatGpt esp32 bot in only a few weeks, and it's fun to load new prompts on it. Wish I had the same memory as the chat interface, that one is pretty good. Now I send a summary with every api request, also increasing token count. Also I don't think a local LLM will be upto par, but the frequency can be real time and unlimited. But that will be the 3rd project, my second bot is with a pi zero.

[@SD-tj5dh]

You need a bumper sensor all around the robot so a signal can come back to saybifbits collided with sonething. Pair that with an enhanced memory, and then it will remember that every time, seeing something at a certain distance will result in a collision and attempt to navigate around it.

[@Pawlixus]

im proud that we have our young skywalker in poland

[@jathomas0910]

I’m watching this high as hell, when she said “let’s head over there to the wall where humans hang out” I nearly died laughing omfg 🤣😂😂🤣😂🤣😂🤣😂💀🙏🏾😇 12:00

[@KJLT20]

It needs memory and you should add lidar too

[@thomasschon]

The robot started testing your patience, so you decided to end the experiment before it was too late. Dude, you seriously need to attend some anger management classes. 😅

[@g0ldER]

I have a bit called Cozmo, who has some basic internal 3d modeling, it uses this for object permanence! You should try this, it only uses one really bad camera.

[@64jcl]

Using Llava model image descriptions alone is not really enough for navigation although it is an interesting experiment. A thing you should try is to make your robot scan the environment by rotating it 90 degrees, take a picture, analysis, and repeat that. When you have 4 descriptions you can make a judgement about where to go based on whatever the goal is. Ofc this is somewhat slow though. Also you could run the image through a depth analysis model. That spits out a gradient image based on depth estimation and those are very good at knowing where there best path might be taken, although you'd have to calculate approximate rotation based on what area of the image you decided the robot should navigate to (either towards closest object or where there are no objects).

[@MrBooks36]

you could make the the robot have a sense of surrounding by adding distance senses and feeding the info to the llm

[@MrMcBauer]

You should try two LLM´s on one robot. One for decision making and one for the controls.

[@stumcconnel]

This is so damn cool, what a huge step up to have everything local! From that part at around 13:50 where you omitted all the extra output processing and just let it run around, operating immediately on each result, it looks like it never really paused in its movement, but was processing an image roughly every second? Maybe the images were all just blurry because it was moving and couldn't be processed well? Or did it pause briefly to get each shot? Sorry if you'd already accounted for that, maybe the camera frame rate or whatever is plenty fast enough!

[@loskubalos]

No wydaje się ciekawie musze obejrzeć kiedy będę miał wolną chwilę

[@jakub38200]

tu jest więcej polaków niż myślisz

[@loskubalos]

@jakub38200 wiem bo Nikodem jest z Polski ja oglądam jego dwa kanały

[@angelbar]

You need to generate volumetric data from stereoscopic images and create a semi-permanent map with constatn updates

[@bananabuilder2248]

Just a suggestion, what if you added LIDAR to improve obstacle avoidance!

[@FelipePereira010]

😃🤝🫂TEM dublagem para o português, obrigado meu amigo ❤, ganhou mais um inscrito 👍, excelente trabalho!!! 🎉

[@legendaryboyyash]

damn I was actually learning robotics just to make this exact same robot controlled with ollama after watching your chatgpt one, I was planning on using raspberry pi 5 for everything, guess u beat me to it and made it even better lol

[@nikodembartnik]

Thank you! Keep working on your project and make it better than I did!

[@legendaryboyyash]

thanks :D I'll try my best to meet your expectations :D @@nikodembartnik

[@CheerApp]

Great project, just a thought...maybe initially the robot should detect types of objects and based of type "book" found then attempt to read the cover title?

[@MSworldvlog-mr4rs]

You should use ultrasonic sensor in all directions, and you can send a depth map with img to this llm

[@unknownumar243]

U can use hugging face text to voice model for converting text to voice

[@pitong1989]

have you thought about giving the robot proximity sensors or lidar to allow it to see the distance to given objects? It could integrate vision with distance detection to enable faster movement and automating it to some extent

[@GlobalScienceNetwork]

Cool video. Bluetooth latency: ~100-200ms. Just a heads up that this could be one of the issues for real-time obstacle avoidance. WiFi latency: 15-30ms Analog RF systems: ~5-10ms. These systems should be all on board or analog if sending to an extra source for computing. The LLM computing will add further delay but should be quick if you use a trained network. However, if you want to train based on your environment from its sensors perspective I would think you would want to do some training and have a custom network. I am not sure how difficult that would be to achieve. Personally, I am going to try a more basic approach and stay analog for everything and not use an LLM. So it might take me more than 10 minutes to program.

[@_taki.debil_]

You can also use piper tts, it runs locally and you can train custom voice for it.

[@rhadiem]

Also xtts v2, F5, more

[@MrJohnboyofsj]

Build a Minecraft villager, give it the same pathfinding AI as in Minecraft, you'd just need a way of 3d scanning the room into voxels.

[@Honzo64]

Regarding the sound output: why don't you just connect a bluetooth speaker? In case the Jetson doesn't have bluetooth you can add a dongle.

[@jtreg]

so good! Messing about with my Tesla K80 today... a bit limited what it can run but llava is ok np

[@conorstewart2214]

The new jetson orin nano is not really a new model at all, it is just an extra 25 W power mode. You can just update an old jetson orin nano to get this power mode too.

[@OZtwo]

Very very cool! I been waiting for someone to try this! I stopped playing with my robots when LLMs came out knowing it would be better than DL -- was I right? Please prove me right! :) Also mixing both Pi and Jetson you get much better overall servo control as the Jetson really has no power to support them. Very cool! Hint: I hope you use two LLMs that can talk to each other as our brain works... (edit: are you using the LLMs API or directly chatting to it?)

[@stony42069]

Seems turning speed is faster than computational speed

[@alexany4619]

Great, but I couldn't see in the video how you integrated the GPU on the robot?

[@haithem8906]

Try the Kokoro model. It sounds really good and is local

[@michaelpaine9184]

Multi-agent system would help separate the driver (pathfinding) and the decision maker.

[@ChristiaanNdoro]

is it possible to have a separate LLM that will be evaluating how well the robot is doing and prompting it to try again

[@lakshaynz]

👏👏👏👏 excellent, inspiring!

[@truthtoad]

maybe adding a flir cam can assist it's navigation, great work. I want a nano😝

[@redthunder6183]

if your using ollama, you should try setting the context window to be bigger. the default is 2,048 which fills up very fast after 2-3 api-calls especially with images. If ur using a GPU like the 4060ti, you can bumb that up to at least 16,000 easily while still having the same performance. I have 12GB vram, and I am able to run llama3.1 8b with 28k context size for a comparison. This should help significantly for things that require more than 3 steps, you can also keep track of the prompt size as it builds up to know when it overflows and starts to truncate it and forget stuff. Also as for navigation, the LLM has no context of its position, where it is reletive to the world, etc. you would need to design a system to give it enough information to be able to gauge its relative position to make informed decisions. for example, if you are able to get the relative position of 3 random points, it should be possible to triangulate you exact relative position, and you could overlay those 3 points as 3 different colored dots on the image. this is a bad example cuz your asking the LLM to triangulate its own position, but it shows the idea of modifying the image to put it into context more.

[@maglat]

Just use Piper for TTS

[@simeonnnnn]

Zima Blue 💙

[@power_death_drag]

you should add a lidar detector to measure distance seems like it cant tell 5 meters to 10 cms

[@Hojitashima]

what about a 360 camara, where you give it the possibility to see the whole room?

[@sandinopaulguerroncruzatty4440]

Would be interesting a little dron with AI

[@yen6742]

Have you tried to adapt it to a unitree go 2. Or the bipedal robot.

[@5mxg]

Ja czekam na odkurzacz, który będzie miał łapkę którą sobie podniesie kable leżące tam, gdzie chce posprzątać.

[@nikobellic570]

This is the coolest thing

[@christiansrensen3810]

I like your vids great job. But before filming you could have cleaned up a bit. ?

[@imranmaj8159]

thanks great videos

[@5fsrinikshith436]

under 15 mis batch here!!

[@Larimuss]

The problem is the model isn’t trained or fintuned for this application. You’d need to train a model or LORA.

[@MaxSvcks]

Great Project. I'm also trying to build a Robot with a LLM built in. Maybe I want to try to run it completely locally with a raspi and a NPU HAT. If it doesnt work I will self host Ollama on my Server :)

[@erniea5843]

Working with Jetson nano is such a pain in the A

[@nikodembartnik]

Why? So far it seems like any other raspberry pi/Linux based sbc

[@xtraa]

You really bought the musk book? 🤣

[@ryzowskyyryzi]

Stary ja mam dla ciebie mocny respekt

[@HamzaKiShorts]

I love your videos!

[@Dj-Mccullough]

I'd suggest Llava-Llama rather than just llava

[@josh-barth]

Explain the prompts and the data exchanged. How did you form context? You say you're changing the prompt and the task but don't tell us what actually changed. How does the machine go? I get the multimodal RAG aspect, but how does the LLM know to respond with an intent to move? Then how is that given to the Pi? What's that datagram look like?

[@alokrm]

I am also facing issues with jetson orin nano gpio pins, very difficult to identify the right pins to get the motor driver to work. Got any solution?

[@Math2C]

Doesn't the jetson have an i2s interface?

[@narpwa]

skill chair 💀

[@takeraparterer]

"offline" and then it uses paid closed source APIs

[@newmonengineering]

I just got my orin edition a few days ago. I am having issues using the GPIO part but Im not going to bother with it, Im just going to use an arduino or esp32 for that instead and communicate with serial over USB. My last Jetson worked well and I was able to use everything but it was significantly slower than the newer version. My only question for you is: are you using ROS2? Or are you just running your own routines entirely? I ask simply because I know ROS2 is capable but can be a real pain to setup and get working. I am assuming you may not be using it because of the position requirement for motoes. I.E. need encoders for speed and direction feedback. I know there are a few hacks around this so I was wondering if you did some sort of hack to use ROS2. Great job with the robot though. He/she is a pretty neat robot. Thanks.

[@MrBudilaks]

At gpio library, if using jetpack 6.1, have to add model name _super at end model code (for orin nano old/new)

[@newmonengineering]

@MrBudilaks my jetpack 6.1 won't run the python script to configure the GPIO pins. The screen appears and immediately disappears. I'm not sure the issue. I flashed it many different ways, from command line, from sdk manager etc. Its on an nvme drive. I have seen a few people with the issue, tried everything mentioned to try to fix it and it still doesn't work properly. There has to be something that either didn't install properly of isn't right in maybe a config file. But I can't seem to get it to work no matter what I have tried. I spend 3 days trying every forum suggestion and it just doesn't work at all for me.

[@MrBudilaks]

@@newmonengineering check your os sys of your module orin nano code. Mine is 8005 at end because dev kit. Check if gpio already installed. Find gpio library which has compats gpio pin data same as your module code

[@MrBudilaks]

@@newmonengineering then i found another problem beside gpio, and have to find library which is module code have to add _super. Due to my work have to be finishes soon, so i put sdk manager at ubuntu 18.04 and install 5.1.1 on nvme

[@MrBudilaks]

@@newmonengineering all & all, nvidia add word "_super" at firmware 6.1 to work at 25Watt cpu. Library which using model code, have to add _super. Will try again when i get new order AI sensing

[@YashKadam-k5q]

Try running Moondream AI model

[@beastmastern159]

u can use rasberry AI hat whit hailo 8 module to run llama 3.2 module, rasberry 5 have great grafic computing capaciti but the AI module is great for runing IA models, i follow u, i like ur content keep going

[@AmoZ-u7b]

Hy man I love these series

[@miltontavaresinfo]

Very nice 👏🏽 New subscribe!👍🏽

[@alrimvt02]

hubieras usado xtts para la voz mejor, gratis y local, ademas el modelo que deberias usar para el proposito que deseas es gemini flash 2.0 este si cuenta con una vision mucho mas avanzada respecto al modelo que usaste.

[@tsungxu]

could you just use the Orin Nano without the Pi? It can basically do everything the Pi can do right but with much more compute

[@X862go]

Awesome 👌

[@robertpoynton9923]

This would be amazing on your Indy mower! No field wires no GPS. Just then it could avoid obstacles, rocks, 💩 ECT

[@Willie-vr6gk]

Jetson can output good signal (I think at least 16kHz), so why don't you connect to Jetson's output pins the speaker (I think that amplifier isn't really required here)?

[@imdaboythatwheheryeah]

Says he wants a local system, 5mins later uses a payed TTS service because he gets money from them. Your goals are flimsy and disappointing

[@rhadiem]

👌

[@g0ldER]

05:10 you should try using the Intel Arc B580, it has more VRAM than the 4060 (useful for LLMs) and is way cheaper

[@Willie-vr6gk]

But slower (OpenVINO is really slow for now)