Inverse Kinematics Upper Body Model

[@Mystfit]

Was this ever released? This still looks to be one of the better IK upper body implementations I've seen for VR.

[@okreylos]

In a way it has been. It's part of the Vrui VR toolkit. You can see the current version in action here: kzbin.info/www/bejne/eIe3iHmkib2YiKs

[@尚毅博]

vrui toolkit is a unity asset? how ican get it​@@okreylos

[@vk2zay]

I'd be interested in rolling this implementation into our skeletal API for all Steam VR users - if you are?

[@okreylos]

Sounds great! These are early stages, though. There are still some edge cases to sort out, and there's no collision detection between parts of the body model yet. I'd be happy to let you try the current state for yourself. I think I still have your email; I'll send you a separate message.

[@gennoveus]

All this with just head and controller tracking!? Incredible. Best IK implementation for VR I've seen by far. I've been subscribed to you for a few years now and you never cease to amaze me.

[@Sonicspin]

This is superior to what I see in many games, amazing work!

[@MoctorDac]

it's impressive! only thing that irks me is that really aggressive elbow snapping, but apart from that it's looking pretty sweet

[@okreylos]

Yeah, there's some improvement left in how the torso is rotated. That should remove some stress from the elbows when the arms are close to extended.

[@fluxrez]

How does it know, when the controller is rolled that its your elbow and not your wrist?

[@okreylos]

It doesn't. The core of the algorithm is wrist pose relaxation, meaning minimizing the relative rotation between the wrist and the lower arm. There is a weight parameter determining how much of that rotation is taken up by the wrist, and how much by the shoulder (causing elbow rotation). The default weight is 0.5, which splits the difference evenly. That's purely heuristic, based on what looks and feels good.

[@fluxrez]

@@okreylos Fascinating! it works really well.

[@MildMisanthropeMaybeMassive]

How long before we see this innovation ported into current games?

[@Uejji]

This is really good. How do you determine the position and orientation of the torso from the limited number of inputs?

[@okreylos]

Heuristics. The horizontal shoulder angle is derived from the pointing directions of the hands, and the torso pitch angle is a function of the head's elevation above the floor, assuming that if you stand up straight the torso is upright, and if you lower yourself down, it rotates forward.

[@Uejji]

Fascinating. I imagine introducing foot tracking could provide an even more accurate full body simulation.

[@okreylos]

I do have one Vive tracker, and I'm going to try how that would work once I'm happy with the default upper body behavior.

[@lampertc1]

I am new to using tracking cameras or mm wave technology to overlay a feature onto a person's image in a video and would like to learn more. I know this video isn't exactly this as you are using tracking data from a VR headset and 2 hand controller tracking data to overlay the upper body with your armor shell graphic. But it is similar enough that people watching this might know something about my specific interest. Would you recommend any videos/books or lectures on this subject to start to learn about using physical location technologies (camera or mm wave radar based or other) along with a video image to do this kind of overlay of a feature onto a person's image like is done in this video?

[@bassemb]

Looks good! Better than some games I've played where the arms completely freak out during certain movements. At 1:03 when you tilt your head to the side, the model thinks you're moving. Can you make it so the model recognises the tilt of the headset and prevents moving the virtual head/torso?

[@MoctorDac]

I feel like that'd be pretty hard without waist tracking, how would it know when you are and aren't moving your torso in those cases

[@SamMotioneer]

I noticed that the virtual arms are the exact same length as the real arms. is this automatically (or manually) calibrated, or just hard coded to fit you?

[@okreylos]

There's a calibration procedure that measures: 1) wrist position relative to controllers 2) length of lower arm 3) shoulder position relative to head 4) length of upper arm. The user has to go through a sequence of motions, and the software derives those parameters through non-linear fitting. The avatar, on the other hand, was hand-generated by me for the measurements taken by the software. It wouldn't be difficult to create a standard avatar and adjust it automatically, but I haven't gotten started on that yet, and 3D modeling isn't really my strong suit (as should be quite obvious).

[@SamMotioneer]

@@okreylos very fancy! If you have the chance, a video on what poses you use for the calibration would be really interesting.

[@LostMyEd]

This should be integrated into all SDK's for everyone to use - obviously with yourselves handsomely rewarded by the manufacturers /platform providers.

[@eobet]

So what will happen with this? Was this just an exercise to be put in a drawer and never seen again, or will it actually be released and benefit the VR developer community?

[@zeejenkins]

Would you be willing to share your source code? Im trying to implement this in Unity following the steps in the paper, however, I am having some troubles with the elbow rotation and love to see how you implemented this. Thank you, Zee

[@okreylos]

I'm currently working on integrating IK avatars into the new version of my remote collaboration framework. It's all open source, so it'll be released soon. Unfortunately I can't set aside the time right now to package it separately as I'm under a tight project deadline. I had the same issue you ran into: the description in the paper, especially of elbow positioning, was too vague to re-implement. I did a couple of different versions by trying to read between the lines, but nothing worked and I ended up ditching the paper's method and rolling my own, following the basic idea from the paper. I also ditched the paper's Euler angle representation and replaced it with quaternions, which removed a lot of the special cases and special handling that would otherwise have been necessary. That's the system shown in the video here.

[@zeejenkins]

@@okreylos Awesome, looking forward to it. Wish you luck on your project!

[@jonasmlgaard8884]

@@okreylos very nice work. Im also trying to do my own implementation. is this implemented in Unity as well or UE4? Any source code you can/will share, would be very much appreciated.

[@okreylos]

@@jonasmlgaard8884 This is built on the Vrui VR toolkit. I got delayed on another project, but I will get back to this soonish and integrate it into the Vrui collaboration framework, at which point it will be released.

[@viniciusnoyoutube]

Impressive!

[@TheSnufking]

Next step: Inverse Kinematic legs?

[@okreylos]

That'd be pure guesswork. The hands are holding trackers at least, but there's nothing on the waist, let alone legs or feet. The main idea right now is how far one can get with only the standard set of trackers.

[@w00tazn]

Would it be possible to use the sub mm tracking to measure the distances between the shoulders and the lengths of the shoulder to elbow / elbow to hand to have even more accurate tracking? I keep wondering if this would help solve any problems but I haven't seen anyone implement such a solution

[@okreylos]

That's what's happening here. There's a calibration step, which is not shown in the video, where the user performs a sequence of motions, from which the software determines 1) wrist position relative to controller, 2) lower arm length, 3) upper arm length, 4) shoulder position relative to neck, 3) neck position relative to head tracker.

[@EZCarnivore]

I think you need to work on your archery form a bit, shouldn't be leaning back at all. Really awesome IK implementation though, I'm sure it'll make you some money!

[@NSPlayer]

so why hasn't gabe newell or john carmack hired you yet, quit your uni projects sir[