Using an LLM to direct our robot Digit.

[@talltomtube]

It cleaned the mess without saying it was going to go have a smoke break first. It's already better than some of my past coworkers.

[@kinanlaham744]

It cleaned the mess without complaining about its previous coworkers.

[@metaphysicalArtist]

Fantastic!! Wow, this is such a great idea! I would love to see a video about how the robot recharges its battery and interacts with the dock. It must be so cool to watch it go in and out of the station.

[@blackhawksp4453]

I just noticed the new legs, even tho I saw the previous promat videos, I find it very cool to see a design go throug streamlining and optimization.

[@lazarusblackwell6988]

Amazing work Agility

[@os3ujziC]

Hey, Digit! Execute Order 66.

[@theonlythingihavetosayis9333]

*HAPPILY*

[@dgoodall6468]

Do it

[@joseventura9685]

Digit says:processing processing waiting... Waiting... robot will begin countdown self destruction mode mode on. 10, 9, 8, 7, good bye 🤖🤖🤖💣💥💣💥😆😆😆

[@Nine-zz6cs]

it will turn the 6 6 upside down like i is :)

[@microMobilidade]

Wow so you can program them to participate in a fire drill! If the alarm sounds they have to go to an exit thats not close to the fire and to avoid humans. Or they could lag a bit to verify noone is left in the building calling for rescue. You guys are great!

[@kmo475]

Amazing, this is what has been missing in robotics so far, a brain. It feels like this plus the hability to speak to ask questions, an internal state and some memory would pretty much give you a working generalist robot that would actually be hable to do a lot of jobs.

[@cuthbertallgood7781]

The devil is always in the details. The LLM is providing a good high-level set of commands, but we don't know how general the robot is at navigation, object identification and manipulation/coordination. The latter is the hard problem. Tying in an LLM is cool, but doesn't tell us how for along they are with the hard problem. It certainly worked for this specific video take and this specific set of objects.

[@kmo475]

@@cuthbertallgood7781 I personally think the hard part is the LLM, providing actual reasoning an intelligence for the system.

[@tahsincankose]

​@@kmo475 Manipulation and object identification are categorically much more complex problems than high-level task planning which LLMs replace in this video. Actual reasoning and intelligence are very vague descriptions. What we see in this video is the mapping of the objects (let's say O_1 to O_n) to bins (B_1 to B_n). The commentator already states that all these are input to the demonstration in the video. LLM does nothing more than relating those O_i to B_j using natural language primitives embedded in the language model. The same can be done with a very simple data structure (e.g. map) where the types of O_i can be related to B_j. All the rest would still have to be done by lower-level capabilities such as localization, navigation, manipulation, motion planning, and object recognition. Having said above, I'm not saying that LLMs are useless. Such a data structure would become intractable in the real world for a generalist robot, as you suggested as well. That's the place where LLMs offer real value. But we won't see human-level androids around us until they have the same lower-level abilities as humans do.

[@Kyle-us6pk]

@@tahsincankosewhat if each robot is only programmed to keep its immediate area in good standing. So the LLM is there to be an interface between man and machine, and anything the human commands will be expected because not only will the AI have the LLM to interpret the commands, it will have ML to help it remember how things should be and how to get them back in good standing. Also, programming each bot to a different portion of the factory would be relatively easy compared to teslas approach of teaching Optimus the same sort of object detection as the self driving neural nets.

[@Withnail1969]

It's all fake.

[@first-thoughtgiver-of-will2456]

Excellent utilization of next gen tech and very innovative! Keep up the awesome work!

[@phmiii]

Amazing work! We have come so far.

[@808GT]

Well done Agility Robotics team! I saw you 2 years ago in the UK at Goodwood and the progress you made in HW and now with LLM integration... Bravo! You are literally eating Teslas lunch atm. Hope you guys find enough funding to go big. Whens the IPO?

[@ianb3053]

Digit LOVES work!

[@chenwilliam5176]

Amazing ❤ But it's not enough for a family 😊 We need a 'house keeper Robot' replacing the female housekeeper we have hired and worked several years ❤

[@cpu6850]

First operational robot, congrats

[@witherwolf3316]

Interesting choice to go with digitigrade legs, makes for a very unique design

[@lucasantonio5769]

wonderful! I can't wait until these little beauties can live in our homes and help us with the housework!

[@autingo6583]

Excellent! Now it's getting REALLY interesting. :-D

[@AWESOMEVIDESHEE]

Agility Robotics gooood jooob, ai robots in for people and world ))

[@davidtro1186]

How does it discriminate between materials? Known objects? Awesome demo! Please show more!

[@Withnail1969]

It's just a fake video. Hidden in the small print will be the fact that the robot was under remote control by a human.

[@JJs_playground]

This is absolutely amazing. Wow. We are getting closer and closer to general purpose humanoid robots.

[@datalifter]

You guys are doing great! it's so fun to watch your systems progress. Keep up the great work.

[@Sirmenonottwo]

If you could just get Digit to walk around like it does not have to pee now that would be great! Or rename it Figgit for more accuracy.

[@Wanderer2035]

We need more updates like this showing how the robots are progressing. Not giving us a video update like once every 2 years

[@donaldsunny7836]

The Digit robot puts the newborn baby with the dirty nappy in the waste bin.😂

[@DominicI1]

This will be the future. If you don't mind me asking, which LLM is being used?

[@SeanKula]

It says #gpt in the title

[@DominicI1]

​@@SeanKula I understand that, but there is GPT-3 , GPT3.5 turbo and GPT4. These are just generalized, there are more specific instances of these model sets.

[@SeanKula]

@@DominicI1 scroll down in the comments somebody asked and answered 3.5

[@AgilityRobotics]

3.5

[@DominicI1]

@@AgilityRobotics Thx 👍

[@JJs_playground]

Would you make digit move faster?

[@ProlificSwan]

Is Digit doing general image classification to determine which type of object it is picking up and what to do with it or are these known objects that it has already trained on / is the trash in known positions with known objects?

[@AgilityRobotics]

No. For this test, we seeded the world model with semantics in advance. But scene labeling from a vision pipeline is reasonably well understood.

[@ProlificSwan]

@@AgilityRobotics that makes sense! I suppose the harder problem in this context is grasping in general, which Digit has shown to be reasonably consistent at.

[@jackieclan815]

This is cool stuff!

[@persianguy1524]

What happened to his eyes and mouth? Hope you guys keep those at it made it more humane and relatable.

[@pile333]

Amazing. Will they be as fast as a human in a few years?

[@AgilityRobotics]

For the tote manipulating video from Promat, we're already about 2x human throughput. The apparent "slowness" is fascinating! What we think is happening is that the robot works super consistently, which results in a higher average speed, even if peak speed is lower.

[@Pyseph]

​@@AgilityRobotics That's incredibly fascinating. I always imagined speed to have been a limiting factor as processing the data would take time, but it seems like we've already reached past that point!

[@pile333]

@@AgilityRobotics Great job. Yes, the average work would be surely higher, but I wonder if they will soon be able to be faster than a human in order to be more efficient in a dangerous emergency scenario.

[@chenwilliam5176]

I hope Digit Robot will be able to drive a car for it's human master in several years. If it can be implemented and commercialized, I will purchase one so that it will be my private driver and drive my petrol car and It's not necessary for me to purchase a 'real L5' Full Self-Driving electric car ❤

[@Meta_humane]

I love you Digit!

[@davidanalyst671]

your mom loves digit too.

[@Meta_humane]

@@davidanalyst671 yeah, she better does. Say I love you so that the robot will remember you & not kill you in the robot uprising

[@lookout816]

Excellent!

[@Music_Hacks-now]

1:01 me at work when the boss only gives me one job to not destroy the cardboard

[@Player2blood]

This is amazing

[@Nine-zz6cs]

Nice! it can walk backwards so it dont have to turn and loos energy

[@parkhayoun]

Nice

[@KaletheQuick]

That's nice. Now Star Wars it up a notch please.

[@AgilityRobotics]

Ok. kzbin.info/www/bejne/a5LKi6Whis6ZetU

[@spinningaround]

More like a Grasshopper then a Digit

[@lazarusblackwell6988]

People need to be freed from their work so they can live their life and not just be slaves or cogs in a machine.

[@pandoraeeris7860]

Not bad. I, for one, welcome our new robotic overlords.

[@Srindal4657]

Imagine what this tech will be like in 30 years

[@jacksonvaldez5911]

Unreal. Imagine walking by a construction site and seeing a massive structure being built entirely by humanoid robots.

[@jeffsteyn7174]

Notice little to no cuts between movements. Unlike vapor bots from tesla

[@Potato-qw4ek]

sex robots when?

[@jomfawad9255]

Does openai charge you tokens per task? or just one time purchase?

[@melvingeraldsy1552]

How heavy can it carry?

[@AgilityRobotics]

About 15 kg, give or take. This version was designed for higher loads consistent with unloading tractor trailers, but for most tasks 15 kg is adequate.

[@yeremiahangeles7252]

Don't you think you should make it stronger to work in industrial settings?....

[@jonwatte4293]

"Digit, clean up the kitchen." Worth $100k right there. Where do I sign? ;-)

[@illygah]

the size of the PIP window changes and youtube chose that as the moment to loop in the thumbnail. it looks like the robot is moving more than it is. I thought that it was shoving something very big at first. Then I realized it hadn't moved anything yet, but that the thing looping was very very short in duration and then I finally saw it was the picture in picture. Don't animate that graphic! The only thing moving on the screen should be the robot and the visual representation of the data its generating because otherwise it doesn't communicate the intended message. It's not significantly misleading, but it is a mislead. I got misled. The robot performed better than the thumbnail had made me think it had. I was not going to explore the content behind the thumbail initially, infact I still haven't. The only reason I clicked this link was to leave this comment. My favorite thing about agility robotics is that they're smaller than humans. Makes them seem less threatening. Maybe that's a bad thing in the long run. We've programmed ourselves to feel threatened by the computerized dystopia ever since 2001: A Space Odyssey pushed the thought into the popular consciousness.

[@Will_Huff]

Love this!

[@Galaxia53]

Which LLM does it use?

[@AgilityRobotics]

GPT 3.5

[@Galaxia53]

@@AgilityRobotics Thanks for answering. Microsoft has the most powerful LLMs out of them all but it's closed source and censored making it far worse than it has to be. There are a lot of open source LLMs available too for free. I hope to see an open source variant in a robot at some point because their training data and architecture are completely open giving us the ability to trust them. One of the reasons I don't trust ChatGPT is because the data it collects is seen by OpenAI/Microsoft and shared with their partners. Another benifit of the open source variants is that they can be ran locally. Especially you guys could probably afford to run a big one.

[@notalkguitarampplug-insrev784]

« Muuuum! DIGIT throw my bag in the trash bin!!! »

[@AgilityRobotics]

Sudo take out the trash

[@notalkguitarampplug-insrev784]

@@AgilityRobotics haha great job by the way :)

[@raoultesla2292]

The Bird/Insect leg stucture is 30%+ superior to Boston Dainamics toys. Oh, and your AI is not shabby either.

[@koko969w]

Amazing

[@benzed1618]

oooooooooooooooooooOOOOOOOOOOOOOOOOO where is the voice command to complete the task ??? Chat GPT where is the voice command to complete the task ???? Chat GPT

[@theterminaldave]

My guess is that humanoid robots will start to go mainstream in 5 years or less. Mainstream meaning performing work and tasks normally done by humans, and probably replacing low-skill labor. I bet mid-skill jobs like house painting will be start to be performed by that time.

[@banksuvladimir]

Doubt it

[@akira5872]

That's really cool but I hope Digit's interpretation won't cut people's hand hahaha (but you get the point)

[@martinlentz-nielsen6361]

🔥🔥🔥🔥

[@DoctorNeil]

I think ai plus robotics is a new beginning for the human race science fiction becomes reality.the only thing left now is for a robot hand to be as dextrous and fast as a human .that is the final piece in the puzzle remaining

[@pamelanay2195]

I want one ☝️ color is nice. Amazing

[@kacperxt371]

In my opinion digit is more intelligent than chatgpt

[@meimeileigh8959]

I’m interested to learn about safety precautions. What if you requested that it behave violently in some way? What kind of safe guarding does it have?

[@TheValiantZero]

Actions need to be taken very early to prevent misuse of this. This machinery and code needs to be foundationally coded to make misuse fundamentally incompatible with the design. This means: -The inability for these robots to be used against human beings -The inability for these robots to detect faces or fingerprints -The inability for these robots to make physical contact with a recognized human being The inventor of the first machine gun thought he would be saving human lives. He ended up destroying hundreds of millions. Let's see how well you match up to your predecessors. “It occurred to me that if I could invent a machine - a gun - which could by its rapidity of fire, enable one man to do as much battle duty as a hundred, that it would, to a large extent supersede the necessity of large armies, and consequently, exposure to battle and disease [would] be greatly diminished.” ― Richard Jordan Gatling

[@techraan2160]

Give it a British accent.

[@CaptainSpoonsAlot]

Digit, invade Cuba. By your command

[@AmySteve2008]

Elon musk must be crapping his pants right now.

[@persianguy1524]

His robots not coming out for another 10 years. He loves to hype things up and make empty promises.

[@SirHargreeves]

@@persianguy1524 Sorry he can’t predict the exact day state of the art novel technology will be ready.

[@Datdus92]

This is way too clunky for in bed, bros

[@zaurenstoates7306]

I for one welcome our new robot overlords!

[@mikhailbulgakov1472]

I wonder why people don't talk much about Agility Robotics. It is clearly way ahead of Tesla Optimus but we only hear about Tesla as if everything that Elon Musk touches is golden even though Twitter is a disaster.

[@jhunt5578]

This is nuts 🤯

[@ChristianBlueChimp]

Brave new world.

[@douglasevans-hs2lp]

Good luck telling it to clean up a hoarder house mess.

[@trooper6627]

So wait... it took them two weeks(80hrs) to program the robot to pickup that trash?

[@AgilityRobotics]

Nope, took about 3 seconds. Re-watch the video.

[@Galaxia53]

Large Language Models are capable of much more than traditional programming for specific tasks. LLMs can be very smart

[@mrnobody.4069]

So you made a Tesla bot but cheaper. AMAZING!!

[@guitarazn90210]

If they can get rid of that pesky LiDAR, this could be the first mass produced humanoid robot. Everything else looks pretty economical. Modular hands would increase the economic utility as well.

[@gcmisc.collection45]

It seems obvious that mankind has created a new Species "In the same way that a book can provide a gripping narrative with words and descriptions to invoke a reader, a machine can also provide a spoken or written narrative. Just as a book is inert, so too are algorithms that act as the book. People see faces in clouds or inanimate objects and feel emotions through inanimate words. The body of a book has no sensory apparatus for interactions, nor do algorithms, avatars or black boxes have sensory apparatus for interactions. Millions of years of culture and human conditioning create the images within the mind. While AI creates jpgs, uses synthetic constructed words, text, film format and binary digital information." Without any knowledge of Organic, sensual, chemical, person perceptions. Just like a digital book. (A) Algorithms calculate in a mathematical way O & 1 @the speed of light. (B) Brains work in an organic biological way via evaluation & culture. Two completely different species and mankind will be the inferior. Even with these head bands

[@BenSomersett]

This is a simulation.

[@AgilityRobotics]

Nope. Real robot.

[@paulmaul23]

Mr. Meeseeks v0.1

[@maxnao3756]

There are too many small leg motions that use energy for not much. Can't this be optimized?

[@marshallmcluhan33]

A prompt injection attack on legs.

[@WildEngineering]

my dog also menacingly stands on his rear legs

[@heydude7568]

great cgi

[@kennyfordham6208]

...and it's not even good CGI. Yeah, this is definity fake 🤨.

[@Thrillbo341]

This is amazingly horrifying. Now send these out to hold cameras and microphones, steal all the entertainment jobs teach it to swing hammers and a waft paint brushes, take all of construction and fabrication no need for fallible doctors, we have automated solutions no futher call for carers or caterers or waiters or barmen the end of farmers and fisherman and sculpters and tailors no boys need go off to war, shredded at the whims of who even knows just leave us the warehouses, the shelves, with the boxes protect our ritualistic packing of cheap foreign made plastic distractions

[@lorenzocoluccia8002]

normalize robots that can do a flip

[@ginogarcia8730]

Ah shiz Skynet haha

[@EmptyNonsens]

It seems like the only safe place is gonna be Internet free zones xdd

[@EmptyNonsens]

Yeah, but the starlink xdd

[@davidanalyst671]

you gave the robot a ridiculous command. Clean up this mess is something a woman boss would tell their employee. There was a pack of black styrofoam that was still taped up, and it should have gone on the shelf, but the robot put it in the trash. Those water jugs are not recycled, they are re-filled. The black foam and the water bottle should have been put on the shelf. This is why managers have evolved to be idiots, while the robots and workers are doing everything they can.

[@theonlythingihavetosayis9333]

You guys really don't Interact with your commentators huh?

[@AgilityRobotics]

Nope.

[@AgilityRobotics]

Wait, are we interacting now?

[@AgilityRobotics]

So confused.

[@kaleidophon]

Meh

[@SurvivalSavvvy]

To slow, Can it pick up smaller items that only fingers can. Long ways to go

[@SirHargreeves]

Slow compared to what? This will work consistently for 24 hours a day. Humans work 8 hours, get distracted and slow down.

[@SurvivalSavvvy]

@@SirHargreeves The example is large items. It has no dexterity with fingers at all. Its limited where a human can do many type of jobs. In polaris in MN you will see people using sharp knives and cleaning up a plastic hood from a snowmobile lid that comes out of a mold. Then in 3 hours they move to another station which requires different movements. For the cost of these and the repairs the human is still cheaper and can be used in more departments. Maybe in 10 more years but I see to many limitations with this. The example they provide is a horrible example. I have not seen one example where they put in live test area these are controlled.

[@trackme9553]

Now give it a gun.

[@AgilityRobotics]

Nope. www.bostondynamics.com/open-letter-opposing-weaponization-general-purpose-robots

[@trackme9553]

@@AgilityRobotics I respect the sentiment but somebody is sooner or later going to do this. It is kinda inevitable.