NVIDIA’s AI Looks At 4,000,000 Frames, Learns To Walk!

[@Legacy_Of_2009]

The walking is super natural! Can’t wait to see this type of stuff implemented in video games

[@Qwa7]

And 3d software, like houdini.

[@ref8893]

..and robots!

[@kuromiLayfe]

get ready for 24 GB of VRAM being the minimum requirement just for this ai system to load

[@BabySisZ_VR]

@@kuromiLayfe good thing we'll have 5090s with 32gb!

[@kuromiLayfe]

@@BabySisZ_VR if you are willing to lay down 5 digits for it :D

[@yagelbar]

I don't always copy on tests, but when I do, the guy I am copying from is already two papers down the line 😅

[@therandommusicguy4773]

xD

[@joseperez-ig5yu]

I've been there before!😂

[@pandoraeeris7860]

Before you can walk you have to twitch!

[@TheAkdzyn]

The best motion physics I've seen in video games is football games like fifa or pes. I can only imagine how immersive the movements will be in up and coming games.

[@SP-ny1fk]

1. Put the AI in a desert environment 2. Give it limited resources (wood, stone..) 3. Give it points for staying healthy in the environment which is always trying to deduct health points 4. Watch a civilization grow before your eyes or 5. Watch the AI character die horribly

[@singularity6761]

Watch the AI creating AI

[@Haze_Nexus_real]

yeah thats actually similar to what i was thinking of, which is literally just putting a large complex neural network in minecraft, giving it no prior knowledge of the game, and letting it learn. the way how its rewarded would be interesting though, maybe like points for eating/being at full hunger, lower points for when it loses hunger/health, etc, but im not sure what it would do in the long run, like, it would probably just dig a hole and make a farm or something to survive forever thats where mods come in

[@ustanik9921]

The simplest way would be to reward it for how long it stays alive with points. The digging in part could be solved by not letting plants grow without sunlight, but then again it could just make a glass ceiling. I guess the best way would be to simulate boredom by attributing negative points for idling or doing repetitive tasks​@@Haze_Nexus_real

​@@Nvidia-LoverOr perhaps it'll be more ambitious and those are the things holding us back.

[@MineAnimator]

​@@Nvidia-Lover sentimento de poder é um conceito abstrato

[@Yourname942]

Next paper should incorporate weight/gravity in the clothing/armor and see how the animations react to it (I hate seeing massive cartoonish characters or characters with tons of armor feel weightless)

[@electroncommerce]

Which humanoid robot company will implement this first?

[@danguafer]

I don't think this is intended for robotics. By looking at the video, specially on the training part, it seems like the animation model attempts to match an animation to a pre-defined trajectory.

[@electroncommerce]

@danguafer Train using Diffusion method virtually, using real world physics and physical robot constraints, and once perfect upload to the robot.

[@CHIM3RA.]

You first need to build a robot that is capable of the fluid movement that the AI makes before you implement it. Like if the bot doesn't have the capability of fluid ankle rotations than you won't be able to achieve it that easily.

[@CHIM3RA.]

@@electroncommerce First you gotta build a bot that has the capability to perform such fluid movement before uploading the animations.

[@electroncommerce]

@CHIM3RA. Since the robot manufacturers know the parts and limitations of the components and the overall robot design, these limitations could be programmed into the virtual diffusion training environment.

[@JorgetePanete]

4:00 she went crazy because she couldn't hold onto any

[@robtroman7917]

Pretty soon humanoid robots are going to have this type of agility.

[@JustLocal]

I don’t think so. The hardware is not ready..

[@noone-ld7pt]

@@JustLocal It's not far off. Watch the latest Tesla event, those movements were surprisingly smooth imo! And then look at the raptor bot from KAIST hitting 46kph. That is faster than Usain Bolts world record of 44.72. I think with the investment AI and robotics are seeing these last 2 years we are going to see absolutely amazing things in the next few years!

[@jcridge]

It's intriguing why the hardware is not ready though. I'd imagine that at some scale, it could be. Surely the mechanics of the simulation can translate via nicely into hardware, and the power can be from electric motors or pneumatics.

[@Haze_Nexus_real]

have you heard of boston dynamics?

[@hombacom]

It’s a mistake to think that ai software progress equals to progress of physical machines

[@--Arthur]

I remember when "Overgrowth" was pushing the limits of physics based movement. This is truly next level🤯

[@PotionScape]

It would be interesting if you could have a key frame where you specify the transform for one specific body part in relation to the torso (or relation to the world), for example, the head being turned right to look that way while letting the body do it's thing, or giving hand key frames to attack with a sword

[@louishauger3057]

I really, really love your videos; you're doing great work! For me, the pauses between the words don't need to be as emphasized. Great stuff! I also love your AI focus!

[@rashiro7262]

I wonder when this will become common in video games. This would be huge for indie devs, since it's basically motion capture quality.

[@Cl0udWolf]

this is super impressive, i just wish some of this tech could come over to engines like unreal and blender sooner.

Motion matching...For real. Also they've added muscle deformation with ml

[@Cl0udWolf]

I think the tool that automatically moved the character to fit an end pose is the most useful. You could create a few end poses for places the character has to move (custscene type stuff) and have the character go there dynamically from anywhere.

@@Cl0udWolf Motion matching... does exactly that. You're asking for something that fully exists...

[@Cl0udWolf]

Last I heard, Motion matching pics the best animation to fit a trajectory. This has the character make it’s own animations and trajectory to get into a position at a location. That’s like saying forward is the same as inverse kinematics because they both are kinematics

[@AI-Life-123]

Another excellent video! I learn something new every time I watch your channel. Keep them coming!

[@JonDoe-uq1mk]

Next job on the chopping block: 3d animation

[@SangoProductions213]

Doing it in real time is neat... But that takes processor time. It would be best if this generation was used to fill out keyframes in animation instead, so that it can be pulled up like our more common animation techniques.

[@smolapril]

so instead of controlling a character, you merely suggest it to move, like how we ride horses. 10/10 I'm sold.

[@adriaanb7371]

it does see the head as just a piece of weight instead of looking ahead

[@skunkwar7468]

Can’t wait to see a enemy skip away from me right after shooting me 😂

[@seanrobinson6407]

Why would you want that?

[@Andytlp]

If the cost is some delay in controlling the character to get realistic physics, gamers will get used to it. Its probably just a bit more delayed than rdr2 controls

[@JonathanTippy]

I'm not so sure; the problem is the inconsistency. The responsiveness will vary based on the current position which might make for very rage game like controls.

[@Metapharsical]

Yeah, I get frustrated w animation/hitbox sloppiness. Though this might be cool for Strategy genre or the now popular "walking simulator" genre , hahaha Or, perhaps it can be incorporated into gameplay as many games do have you "steady yourself on a tightrope" ... I love games that have emergent physics puzzles even as a main gameplay element. QWOP for example 🤣

[@echorises]

It depends on the genre to use it realtime, for genres that require instant controls, they can just save the animations and use them like they do right now. The difference will be that the animation team will be able to produce 100 animations instead of 10 in the same time. As for the genres that don't really require instant controls, like in walking simulators or any kind of non-competitive slow-paced game, we still need them to feel like there is no apparent lag. On the note: I have been watching these videos for years now. Yes, many papers. No, zero implementations into actual video games (at least in the sense of real-time generated or physics-based generative animations, I should remind everyone that Euphoria is still the king for some reason). There is either some issue with how these research are licensed (patents and stuff), or this channel just likes to exaggerate. There is one project that I am aware of which is called Motorica AI (in which we can generate simple walking animations with presets) and even their resulting animations are not allowed to be in the games. Possibly meaning that they are trying find a way around some possible licensing issues.

[@amkire65]

Impressive. Not only will it be great for games, but imagine filmmakers who need to create huge realistic battle scenes, etc. Three years ago 90% of what we can now do with AI would have been an unrealistic dream!

[@adenwellsmith6908]

When you look at 1 year olds. They might spend a couple of weeks crawling. Then they pull themselves up to stand on 2 legs. Then within a couple of weeks they are walking. Given some of that is building muscle, it doesn't leave many hours to actually learn to walk. It's very quick. But if you look at learning to speak, its a different matter. Even at 1 they can understand instructions at a basic level. But to talk takes a whole year and more. That's a far more complex learning task.

[@Steamrick]

I wonder how much performance overhead it'd take to implement this in a game? How heavy is the neural network?

[@arthurheuer]

5:15 I guess Christmas came early, since budget Santa's holding onto his papers? Am I the only one who sees this?

[@Cyan37]

Nvidia has had amazing AI for quite some time now. When will these models finally make it into games?

[@echorises]

At this point, I am pretty sure that Nvidia itself is gate-keeping these research to be used in actual game engines. It has been so long since their first AI-Driven realtime calculated physical animation thingy was published. Considering that they are still building the Omniverse, I guess that they will eventually evolve into some sort of everything app for 3D and use these techniques there.

[@r.m8146]

this is actually amazing

[@JonathanTippy]

I'm thinking the controls might end up hard to use, like a rage game, because of the inconsistency in responsiveness.

[@Metapharsical]

😎QWOP are you talking about?

[@thatonecommunist]

watch david rosen's animation bootcamp from gdc, it absolutely won't be a problem with that approach.

[@FrotLopOfficial]

I cant wait to see this in GTA 97 and Sims 523 in 1000 years

GTA 6 or 7.

[@begobolehsjwjangan2359]

bannerlord will need this more than ever to animate the characters

[@TheRealityWarper08]

Npcs are going to get way better

[@viteek.t.2278]

yeah!!

[@johngrey5806]

Now, I understand that the narrator's voice indicates that he's still in the middle stages of working out the smooth movement.

[@GarviHere]

Life on earth has been denoising to create ultimate life form...

[@elpedomasgrande1]

Genuinely curious: why is foot sliding even a problem? Shouldn’t it be solved by inverse kinematics? Isn’t it possible to “fix” the foot? 🤔

[@SeaHorseOo]

As you said "Sign me in for future videogame developpement."...🤨....

[@Husani759]

At 2:00, the model seemed to know capoeira. 😄

[@parazels83]

When will we finally see the animation in games?

[@AhmedZayanHabeeb]

I can see that deadpool is still making maximum effort 😂

[@costarich8029]

When are they training it on Michael Jackson's dance moves? I wanna see a little AI do some moon walking and other iconic MJ dance moves.

[@jespado]

Why are the feet sliding on the ground? How good is the physics simulation behind this?

[@JulianSloman]

there isn't any, physics isn't simulated here, but imitated indirectly via the restraints physics had on the original footage

[@UlyssesDrax]

I can imagine a time the future Dr Zsolnai-Fehér enslaved by AI with a big smile on his face and tears welling in his eyes saying, "What a time to not be alive!"

[@peterkamau2014]

Is there some custom software for modelling these AI games?

[@WhoWatchesVideos]

Given the variable responsiveness, I wonder if the first application for this technique's descendants in games will be in a life sim where they player has limited control, anyway, and the character needs to be able to avoid protruding obstacles or floor clutter while, say, carrying irregularly-shaped objects.

[@RichardServello]

This could be good for crowd sims.

[@UCs6ktlulE5BEeb3vBBOu6DQ]

once they are ready, our demise will be swift

[@makinganoise6028]

Should be a fun tool for helping create military robots

[@CruelCrusader90]

i think it will be a long time before the AI captures the human element to go along with this animation system. except for more dynamic movement, games wont be much different.

[@cloudzero2049]

Walking AI has come so far, wow.

[@amykpop1]

When will we ever see this technology implemented into software that we can actually use? It's always just papers and demo's, but never a live demonstration.

[@SocratesAlexander]

Stop feeding them whiskey.

[@_John_P]

Why is noise used?

[@idwardshadow8404]

What are this fields called ?

[@jcridge]

Boston Dynamics Atlas has amazing performance, but still doesn't seem to have the same sort of fluidity as these simulations. This is a puzzle as Atlas seems to have the power, the rotations and the lever type joints that might allow it to do so. Insight anyone ?

[@theftking]

Did you know that while they were testing this, at one point the AI agents just stared at the screen? Even when they shouldn't have had any notion of where the viewport was, they always somehow knew to look _directly at the camera._ To this day, no one knows why.

[@Georgeous42]

Sounds like a good creepy pasta^^ Any sources that this is true?

[@dinhero21]

doesn't sound real given that: 1. The AI shouldn't know where the camera is 2. The AI had no incentive to look at the camera However, the agent might start out looking at the camera by default, and it's just trying to return to it's initial state? (sounds kinda far fetched)

[@research417]

prolly just some weird bug

[@theftking]

@@Georgeous42nah I just made it up.

[@Llllillilililililillll]

What happens when you upload an anatomical muscular system and give it to the AI so it actually has to calculate how to move each muscle in tandem.... At what point do we just create a full simulation of our own reality where the AI we create has to control individual muscles we give it? What are we? Are we not just a fractal creation of God, mimicking what we are with these AI creations? After all, we are just attempting to simulate our own reality, and make it look, "real". Which begs the question, why do we desire so fundamentally, to copy all the values of laws of physics we assign in these programs, in order to make it indistinguishable from our own, yet have full control over it? What end does that serve? Hint: a scary one.

[@user-pt1kj5uw3b]

But can it hit the griddy?

[@pigeondriver8992]

now add collisions to itself and othrts

[@HussainAhmad-b9u]

is this done in unity game engine I would like to know how to do that because I am a unity game dev myself

[@duckduck-oi8yv]

Now make them swordfight!

[@Antipico]

Next level glitchy NPCs here we come!

[@abderrahimbenmoussa4359]

Looks like drunk trolls 😅 That + chat GPT would make quite interesting pnjs

[@RasmusSchultz]

are we ever going to see any of this tech in actual video games? I feel like we've been seeing videos like this for years now and still nothing in games? I'm starting to suspect most of this tech only actually works in isolated cases and well known environments. does any of this generalize? or is it just fun and interesting experiments to look at?

[@bloodust7356]

Those little added comments always makes me giggle.

[@truebones]

yea, but what do they need me for?

[@ellopropello]

i'd so love if there was a game generated by ai, using such techniques for character movements, but even the levels and dialogs created by an ai, making such a game a complete unique experience

[@xewi60]

put this in unreal and TAKE MY MONEY

[@saiaini2439]

it remind me at Naruto learning Rashen shuriken...🤣

[@expodemita]

But they are really walking or they are imiting the movements to walk? Is there a physic and gravity and muscles or this is only for videogames?

[@olihaus5007]

This is a good question, and concerns my research. Afai understand, this work does state prediction, whereas in robotics you require the action.

[@DeltaJes-co8yu]

Nothing Impressive compared to how a human learns to walk in 10-18 months 😂

[@stevemeisternomic]

Teach it kung-fu and see what it can figure out

[@gianfavero]

Meh. I do those moves any Saturday night 🥴

[@marinomusico5768]

Interesting

[@kingbruhh]

Third comment

[@Zzz-u6t]

IN UNREALE ENGINE WE ARE USING MORE ADVANCE THING TAHN THIS YOUR SHOWING SOME OLD THING TO US . LOOK AT THE UNREAL ENGINE MOTION MACHING SYSTEM

[@echorises]

Lol. Motion matching is just an overly complex system that could only result in having a rich animation system. There is nothing reactive or physical about that, it's only matching trajectory to the already available hundreds of animations. So, actually, you can use these techniques to create the "hundreds of animations" that would eventually make motion matching a viable alternative to use in all projects, instead of only multi-million dolar AAA games. Just look at any motion matched game. All the ledges have either x or 2x height, the ground is mostly flat and you can never have really complex geometry that animations can react to, etc. 15 year old Euphoria engine accomplished more than what motion matching can possibly do.

[@Happy-vz3xe]

😭 I don't get first but great video that's matters the knowledge

[@CA-dg6ou]

Second comment

[@TundeEszlari]

I love your videos, you are a very good KZbinr, can I get a comment heart?🙂

[@maxpilip9277]

fourth comment

[@aleksandrszagorskis3306]

First comment 😅

[@WackyGameEngineer]

first comment on the first comment.

[@Thiago1337]

💩

[@jacksonkelley9774]

Your way of explaining things is terrible, as if it's from someone who has no idea what's really happening, or no depth of knowledge lmao

[@adityadhar1358]

I don't understand the purpose of this, why don't just animate a walking ragdoll lol

[@GurbyTheGreat]

They are learning from scratch, with a defined goal. That's the point, the goal is not a product, it's a proof the methodology works and can be applied elsewhere potentially or expanded on

[@Martin-sr8yb]

Fourth comment