New AI: This Is A Gaming Revolution!

❤️ Check out Lambda here and sign up for their GPU Cloud: lambdalabs.com...
📝 The paper "Categorical Codebook Matching for Embodied Character Controllers" is available here:
github.com/seb...
📝 My paper on simulations that look almost like reality is available for free here:
rdcu.be/cWPfD
Or this is the orig. Nature Physics link with clickable citations:
www.nature.com...
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Alex Balfanz, Alex Haro, B Shang, Benji Rabhan, Gaston Ingaramo, Gordon Child, John Le, Kyle Davis, Loyal Alchemist, Lukas Biewald, Martin, Michael Albrecht, Michael Tedder, Owen Skarpness, Richard Sundvall, Taras Bobrovytsky, Ted Johnson, Thomas Krcmar, Tybie Fitzhugh, Ueli Gallizzi.
If you wish to appear here or pick up other perks, click here: / twominutepapers
My research: cg.tuwien.ac.a...
X/Twitter: / twominutepapers
Thumbnail design: Felícia Zsolnai-Fehér - felicia.hu

[@TheAkdzyn]

I'm genuinely impressed with it's leg motion simulation and how smooth it is. Exciting updates as always. Thank you as always.

[@MattPin]

It is pretty incredible, this will be very helpful in the near future

[@usuario8245]

Which company is devoloping this tech? thanks!

[@epimolophant]

Those are the legs Zuckerberg desperately needed for Meta back then

[@Steamrick]

This would probably really benefit from one more sensor to clip to your belt. Acceleration, hip angle, vibration from steps would make lower body guesses much more accurate without major inconvenience.

[@chekote]

Yup! That’s how Standable on Steam works. It just fills in the blanks of whatever trackers you don’t have.

[@ObeyBunny]

Ring Fitt for the Nintendo Switch did something similar. One of the controllers slid into a little Velcro strap that wrapped around the player's leg.

[@spaceghostcqc2137]

The hip sensor should be part of the compute/battery puck. I don't think they should strap all that to the head anymore - only some sensors and the screens/audio.

[@coolbath8313]

add this with some extra knee or feet sensors, we'll get the perfect embodiment tracker!

[@Hugo-pj4bm]

​@@c4blecdepends on what kind of tracking solution you use. The lowest would be about 300 euros with something like slimeVR trackers.

[@jtjames79]

The metaquest 3 can get a decent view of the legs at least when you're not standing straight up. Certainly what I would consider good enough.

[@gaba023]

@@c4blec What do you mean by "cost"? Money or computation effort? Either way, good question. Right now I use 11 point tracking and it was not cheap to buy. It keeps up with my movement 99% of the time though. I use foot, knees and hip for lower body. You want the tracker on the foot not ankle, so it captures foot twist for movement like the C walk or shuffle or whatever dance move you are doing. This ML/AI is a long way off from full capture.

[@thatgotofinal]

​@@c4blec at this point its not about the cost but accuracy, there is only so much you can do with guessing, like no way this could handle someone doing yoga in VR without extra data about legs - and probably would break much quicker with just unusal doge positions or kicks. But its still amazing solution to just make games feel more responsive/natural without adding much cost and burden, for games where legs are not important but you still want them to be visible.

[@carlpanzram7081]

​@@thatgotofinalI also don't understand why they wouldn't just use two additional, simple sensors for the ankles. Wouldn't that be FAR easier, more accurate AND more useful than just guessing their position?

[@AmbientMood]

Wow, adding this to VRChat would do wonders. Just like networks physics collision with other players brings a whole other level of emersion, i think this would be one of those things that one cannot look back from.

[@hungaro7964]

I was thinking the same

[@styxrakash4639]

So they used neural networks to improve the accuracy of inverse kinematics for VR?

[@mtdfs5147]

Yea this seems like a c- paper. Cool but not groudbreaking

[@JustAPersonalUseBarb]

*and predictions of future movements

[@sebastianjost]

Seems like that's what they did. It's very impressive how well it worked though. It's probably not easy to get this level of accuracy.

[@fen4554]

Sounds more like a system made from the ground up rather than adapted from IK. There is probably still endpoint control to plan the animations, but it doesn't look at bone length at all when planning the motion it seems. The motions aren't rigidly slaved like with IK.

[@mtdfs5147]

@@JustAPersonalUseBarb interpolation forward in time isn't new. I know hacks for videogames that do this. Admittedly as the other guy says this ISNT inverse kinematics but it DEFINITELY follows the same principle.

[@Quicksplained_]

This will be great for online VR gaming imersion, where you can see the other players doing more realistic movements, for example if your playing a shooter and the enemy is crouching, it should look like a more realistic crouch and therefore more immersive for you.

[@kaushalsuvarna5156]

What a time to be embodied 😅

[@kalli4231]

*Aint that a Kinect? 😅*

[@hola_chelo]

Yes! A kinect without the kinect

[@therealpeter2267]

This way you don't need anything to be recording you, just the headset and controllers. No console, no accessories. You could take a walk with this outside and it would still work. You could bring it to a friend's house, put it on and it'll still work.

[@aluurbb]

completely different methods

[@CIinbox]

@@therealpeter2267 Except that you need to feed it with 3 hours of training data i.e. recordings of your motions.

[@0AThijs]

​@@CIinboxStill doesn't record you as in video recording.

[@marvin1245]

We finally have the leg technology for the metaverse, surely this will save it

[@DavidInfinityOfficial]

I remember doing this like a decade ago with my Kinect 360…

[@pushpendersolanki89]

what a time to be alive!

[@dan_loup]

The kinect can do that, however it lacked developers that were willing to actually make games that make full use of the body tracking, instead opting for using the device as a glorified virtual button pusher. "make X pose to press virtual button Y" and that was naturally laggy as you needed to finish the pose to the game to start processing the input.

[@ChiggenWingz]

The video title and thumbnail are appallingly bad for what this video is trying to show off. The content is great and I'm sharing it around to other VR devs, but damn if the title and thumbnail way off the mark :\

[@psyenz8946]

Welcome to the world of "AI News" . Every little thing is revolutionary, stunning, and insane.

[@ahmedsamv3988]

i agree

[@cerichson]

The title is for the algorithm to read, not you. 😀

[@fen4554]

Right now any VRchat fan without the budget for (or faith in) body trackers should be very excited. Plus some of my favorite games like Into The Radius and Blade & Sorcery would seriously benefit from this.

[@3lbios]

they wouldnt cause you cant kick with this

[@shang6158]

I remember getting a Nintendo WIi for the first time. It came with Wii Sports, and at first we'd play bowling and baseball the way they were intended, but eventually we discovered that you could throw the ball with a short flick of the hand rather than swinging your whole arm, and that's what we did. Most of the time you don't want an interface that requires moving your whole body, you want small precise movements.

[@mitchellvandongen1732]

I Just found this channel again after being lost for a year. I don't know why I wasn't subbed before, but I am now. I love your videos. What a time to be alive.

[@LarryPanozzo]

1:33 “That is absolutely insanity.” I concur.

[@mtdfs5147]

It's not. This is basically just slightly better inverse kinematics, nothing revolutionary here.

[@kamikeserpentail3778]

What I don't understand is why you can't just use an external camera or two pointed at the person to generate the model and motions of them. Surely they can get the accuracy up using exactly this kind of predicting net.

[@gaba023]

Of coarse you could. The point of the paper is to show how to do it without anything besides 3 point tracking. There are always tradeoffs. For example, if you used cameras, they would fail in a warehouse sized space or you'd need a dozen cameras.

[@bobgarrish]

Yeah, solutions to this problem that'll work for the market have to do exactly what you're saying: use a commodity sensor to get just enough extra info to clean up the uncertainty in an unsophisticated prediction model.

[@jackburton5085]

During the Friends theme song I always wait for the applause, to clap too. For years now in every video I've been waiting for you to say *"What a time to be alive"* so I can say it too. I don't know, maybe I have OCD but... I enjoy it too much!

[@kronux3831]

Imagine how good this tech will be when applied with a full-body haptic suit, or even just a pair of VR shoes. Our immersive capabilities are gonna be insane in about 10 years from now

[@johnemanuelsson31]

What needs to be done next is to input actual data from lower body, such as camera data from headset and actual trackers. Optional support for mirrors would be great as well.

[@sanketvaria9734]

4:40 if not for gaming then only one other thing that comes to my mind that this could be used for. and I don't have to tell what that is

[@dusty_giraffe2111]

I wonder if it’s possible to use AI to generate the view of a slightly different perspective, so VR games would not need to render an image for both eyes.

[@peremoyaserra2749]

Hm, sounds like standard POV to stereoscopic images could be done. But we would be dealing with exactly the same issues we see in DLSS (spatial/temporal stability and disocclusion artifacts)

[@djhindsight]

You don't need to. There is a feature called "instanced stereo" it basically re-projects one eye's render onto the depth map of the other.

[@thirien59]

It’s a funny fit, especially for games where you actually care about watching other players moving But for most vr games, you only care about what you do with your arms, and never about legs movement you cannot deduce from the arms

[@cddelgado]

04:36 I came for the information. I smiled for the moment.

[@NicolasSilvaVasault]

this is peak motion capture !!! i just hope this will end up on meta quest, and not just a paper

[@McDaniel77]

Nintendo did unbelievable things with the Wii in 2006 with Wii Sports! And in 2009 they surpassed that with Wii Sports Resort. Real time motion control with 60 fps.

[@7415_Gamer]

Kinect is back.

[@grey_north9016]

I've been following this tech for a while and it's amazing how much it has been advancing. This will help us get rid of expensive and cumbersome body tracking hardware.

[@jurgenkranenburg1309]

can't imagine how good this will be when you add just a few extra sensors, like strap on sensors on the legs.

[@artman40]

It's not just Dance Dance Evolution! Now you can move your body freely!

[@ironknight413]

I’m very interested in using realistic movement of virtual avatars to develop, train, and test gait and activity detection algorithms. I think with a little more work you could put a virtual IMU on this avatar and generate a ton of really interesting, high fidelity data!

[@RellisLCT]

this would be awesome for a game like Boneworks or Half Life Alyx. can't wait to see what cool games people make from this!

[@YOEL_44]

We have achieved the ultimate walking simulator!!!!

[@casualintrovert207]

I just want to see all this tech added as a driver update for my gpu or for my oculus or SOMETHING 😭 so much useful performance and experience enhancing tech and it takes so long for it to get introduced into games.

[@tiagotiagot]

Reminds me of that paper from over a decade ago, "Style-based Inverse Kinematics". I'm surprised it took this long for something similar to get implemented in VR...

[@TheKwiatek]

Finally, a video about something new without using irrelevant clips from other vaguely relevant papers. Grrat job 👍

[@avamomoh]

Microsft's xbox kinect was simply ahead of it's time. The technology just had to catch up

[@dan_loup]

The tech was mostly fine, its the game developers that had no clue on how to use it. They probably still don't

[@goomyman23]

i wish quest would include a waist tracker - because i get extremly sick if my waist isnt facing forward when i press forward. Maybe this could help if it was accurate enough. Forward should go waste direction. Maybe if the head tracker looked down it could detect that but then if it could do that it could detect the legs too.

[@taichikitty]

Not sure why this is so revolutionary. This is what humans, and presumably other mammals, do while processing high speed events in close quarters. It is one of the reasons we have a section of our brains devoted to judging if a pose of another person is possible or repugnant. At 100Hz, the human brain cannot react in "real time" based on what is present in the sensory information of the "moment"; it has to have one or more "threads" of predictive movement for other actors in the immediate environs. For instance, while doing high speed and potentially complex martial movements, especially in close quarters, the person acting will often have their view of both their opponent's and their own lower body obscured, yet the experienced practitioner will automatically know how to move their lower bodies to deal with the opponents' lower body motion. Time and again this strange obsession with treating the human visual system as a view of the world in full resolution and seamless motion in real time causes these cognitive biases in how human perception actually works. I expect in the not too distant future someone is going to get the bright idea of taking a series of narrow, high-resolution still images*, along with very low resolution but much broader images of the same perspective, but with any uncompensated changes highlighted to indicate independent motion, all from the same point with some variation in direction and focal distance. These images could then be combined in a well trained AI to produce a 3D model of the environment in the field of view around the scanned-from point. That would mimic how humans see. Then the next step would be to do all directions sampling in the same way and you could get a 3D model of the entire environment from the view of that central point. Then move the scanned-from point around and get even more detail. The pre training would involve giving the AI "expectations" of what the "other side" of things looks like based on the side you can see. In humans, some of that expectation is what you just saw a few frames ago before the person turned their face, what you noticed on a side of an object you came around, etc. * Seems pointless to make the images upside-down/mirror to mimic what gets to the retina, since we have shown the human brain has a system for flipping images to correlate perception to the experienced world anyway.

[@neuplop]

I was confused because there are already better things with trackers, and saw something similar some months ago for vrchat that without trackers aproximated the legs really well but still couldn't dance. Guess it made it into a paper. I don't think it will be a revolution, after all is just so third person in vr looks better, when in vr you'll most likely be playing on first person, so mostly vrchat without tracking on the legs is the most significant usecase.

[@jensschmidt3687]

Best 'body guessing' so far and by a fair amount too 🤩 What a time to be alive 🥰

[@bennyboo25]

Meta needs to buy this and implement asap

[@genebeidl4011]

Nothing more evil than bunny boxes 😀

[@Happy-vz3xe]

Man u are blessing to us❤❤ What a time to be alive

[@test-uy4vc]

What an AI time to be rendered alive! 🎉

[@matandessaur]

I want to move in an intergalactic something !

[@callibor3119]

Because the lower half is the same model AI as the self progressing AI with thousands of models moving in different ways. And with a library of different animation assets that that model is calculating, it is like an advanced hangman as the real person’s upper body motion is being translated into actual data with the many animation assets. It’s a slow process keeping up with the real upper body, but it is moving in real-time. It almost looks motion capture like, but the lower body shifts in a way that isn’t true in the real world. But that can be advantageous for privacy and security reasons. It shouldn’t need to replicate one to one every movement the real legs move. It should keep making calculations with the many animation assets that it has.

[@Shaostoul]

Loving this! I'll be likely using it or a future version of this in the development of Project Universe, a sci-fi single-player to MMO scale RPG geared towards ending poverty.

[@besknighter]

Now I want a paper that compares the accuracy of the estimated pose with a really diverse set of trackers, vs the IKs already on the market. HMD + 2 controllers HMD + hand tracking HMD + 1 controller + 1 hand tracking HMD + 2 controllers with hand tracking HMD + 2 controllers + 1 hip tracker HMD + 2 controllers + 2 knee trackers HMD + 2 controllers + 2 ankle trackers HMD + 2 controllers + 1 hip tracker + 2 feet tracker HMD + 2 controllers + 2 elbow trackers + 1 hip tracker + 2 knee trackers + 2 feet trackers Imagine the default (first on this list) scenario using this technique being more precise than the last one with state-of-the-art VR pose estimating IK library that there is?

[@aaronsmyth7943]

They'll need to design new padded walls too.

[@ovoj]

3 hours today, 3 mins in 1 year, 3 seconds in 5 years

[@seanmurphy6481]

I like how it demonstrates how you can move your lower body when sitting down while using your controllers.

[@shadowdragon3521]

I remember a paper from a while back that used WiFi signals to map out the position and poses of humans. I'm not sure why people aren't looking into using that for creating immersive VR avatars.

[@sebastianjost]

Maybe privacy concerns if this tech gets too much attention?

[@fishygaming793]

I don't get why we don't use camera's: it's simple, cheap and very good with modern technology's. (Even a simple 2d camera will do)

[@bennyboiii1196]

@@fishygaming793 The camera stuff is hard. After trying several solutions, there was one that looked promising-ish (Mocap4all), but required a lot of setup and perfect lighting conditions to work, and even then it wasn't all that accurate, even with multiple cameras placed around the room. It's also expensive software. There were other solutions, but they were way worse. Maybe pose estimation has since progressed, but i haven't heard much.

[@julian1000]

Almost certainly accuracy and noise. Similar technologies are currently employed for human tracking in grocery stores, stadiums, etc.

[@Denis.Kochetov]

Another fabulous work from the Starke brothers. I admire these guys; they make the best neural networks for 3D animations compatible with real game engines. Absolute beasts. Huge win for Meta for recruiting them!

[@MikevomMars]

The explanation is a bit confusing. I don't get what this is about - is it about POSITION INPUT from VR controllers and headset and rendering a 3d movement according to this motion?

[@randomarsh9817]

Peppers?! WE DONT HAVE ANY PEPPERS!

[@Legnog822]

hey, even if it doesn't look the same, this would still be increadible for something like VR chat!

[@kellymoses8566]

I actually felt that this would be possible, just incredibly hard.

[@magnumizer]

I mean it looks cool, but Oculus is already doing this with their Movement SDK, using a trained AI + IK. And it's far easier to setup.

[@aftsfm]

No way, we can now realistically sit down and maybe even lay in vr?

[@peremoyaserra2749]

Soo... this is motion matching but for VR? (ore any usecases where we have the movement data of particular body parts I guess)

[@markonfilms]

This is awesome! It's amazing it's this effective. Epic precision.

[@tachrayonic2982]

I wonder how performant this system is, and how easy it would be to implement into an existing game. How well does it manage additional trackers on the feet and waist? Can it deal with additional forces against the body? (Such as in a Boneworks/Bonelab or Blade and Sorcery environment) Can it deal with non-humanoid avatars? It's great to see that the shoulders will point forwards to give the arm extra reach when it is needed, I've seen IK systems fail to implement this many times before. Usually when you reach forward that far, the IK simply fails to move the hand that far forward.

[@pablodm9]

Fight movement prediction googles gonna go crazy

[@NSXtacy-]

That is what made Echo VR and Lone Echo so damn good, for having been released 7 years ago they did a magnificent job making your VR avatar body behave like you think it should. Even 7 years later I don't know if there is another VR game that has done a better job. The inverse kinematics are just sublime, of course zero-g kinda helps achieve that! And Meta was dumb enough to kill the game, still angry about that.

[@rijnhartman8549]

This is great, but... It allows AI to be able to predict human movements, which could lead to something scary one day

[@blackshard641]

You're saying they finally fixed Kinect?

[@gordogonk8068]

I might be missing something but aside from the improved leg estimations, i don't think anything else really mentioned is new

[@fruitcake4910]

Why can’t something like sensor fusion improve the accuracy of leg movements? You know this fused with Kinect?

[@TothNicolae]

meta should add foot sensors for proper mma simulations

[@eldoprano]

Would be interesting to combine with camera pose data using a kalman filter

[@janschneider8647]

we had a kinect that was way more accurate and while I loved playing with it, it seems that it was a commercial failure.

[@MrSoulast]

Looks like motion matching for VR.

[@BaseRealityVR]

I had this idea a few weeks ago about using Motion matching with My VR system :) good to see I was spot on

[@jonathan10007]

ideas are just ideas

[@TheDigitalZero]

I'm so excited that we might get total immersion within 20 years.

[@EtherealEngine01]

military drones predicting body movements to aim.

[@maythesciencebewithyou]

The Kinect was way ahead of its time

[@sebastianjost]

I still prefer the 360 kinect over current VR headsets. Much more convenient without the need for a warm headset and controllers to limit your movements.

[@maythesciencebewithyou]

@@sebastianjost Yeah, I guess I should rephrase it to, "The Kinect was and still is way ahead of its time". I'm upset that my 360 broke down last year. I liked to occasionally play some kinect games. Much better than wii or switch. I hope for a kinect comeback

[@sebastianjost]

@@maythesciencebewithyou just look for a used one they're often below 100€ including some games. If you're quick, you can even still buy some games digitally, but that store will shut down on July 29th

[@Bittermandl]

Can't wait to see something like this with martial arts (instruction).

[@riufq]

0:43 Are you talking the previous technique or both?

[@filippe999]

VRCHAT just got an upgrade

[@chekote]

More like an alternative to Standable, which has been around for ages and does exactly this.

[@nortonwedge]

Honestly, anyone who wants this has never tried playing VR, or even just Wii for an hour or more. Who wants to hold their arms out in front of them pretending to shoot, when you can just use a controller or MnK? It's fun for like 10-15min before you get too tired. This isn't the future...

[@Paulo_Dirac]

Mortal combat is about to get a lot more hardcore... Don't know how you translate that to RTS game though...

[@sadshed4585]

It would be cool to see this in blade and sorcery or vrchat wondering if devs are already working on it lmk in these comments

[@NotYoAverage0]

I am impressed. I like it.

[@parthasarathyvenkatadri]

Next we need neural feedback for gaming and simulations

[@zeekjones1]

So... Standable, but with extra steps.

[@HansMilling]

Many Meta games, lack the lower body for the Avatar, would be cool to see people walk around instead of these floating torsos.

[@Zonca2]

I feel like they should focus on some sort of gloves for the next gen VR, ideal would be gloves with feedback 😅not sure if that's possible yet.

[@HedgehogGolf]

Is that 3 hours of training data total? Or 3 hours of training data for each person?

[@robo1540]

VRCHAT!! ADD THIS AND MY LIFE IS YOURS!!

[@juhor.7594]

This is very dope and a very good example of machine learning put to practice.

[@simian3455]

this feel like i'm watching VR Euphoria engine...

[@baaaaaaaaaaaaaaaaaaaaaaam]

zuckerberg will finally have legs 😂

[@Jai_Lopez]

To be honest I am more mind-blown that you are mind-blowing I expected you to at least know that was possible as she was breaking down this new video in his capabilities off the program I remember a couple of papers ago where some of these techniques we're just starting an even you ask imagine a couple of papers down the line well my fellow scholar this is the paper you was talking about😂

[@cooluke29]

The thumbnail is very close to a photo in letsgameitouts newest video on the in game monitor

[@cortexcarvalho9423]

Next game: x-men danger room simulator!!!!

[@Solizeus]

That is really cool, but doesn't characters in VRchat already move like that?

[@khalatelomara]

it is actually good for cheap mocap somehow 😀