Capturing Detailed Deformations of Moving Human Bodies | Siggraph 2021

The video demonstrating our recently published paper: Capturing Detailed Deformations of Moving Human Bodies.
Project page: ankachan.github.io/Projects/M...

[@lllcinematography]

Amazing job. Most detailed and natural motion capture I've ever seen.

[@myelinsheathxd]

Amazing deformation sense in simulated env!

[@skoklater_rogue]

Has the development of the checkerboard motion capture system progressed? I'm eager to utilize it! As a visual effects artist and game developer, this system could greatly benefit me, potentially serving as a cost-effective multi-camera motion capture solution (targeting 3-4 cameras) with natural lighting conditions. Its intricate design suggests it may rival or even surpass modern IR motion capture systems due to its self-occlusion handling, subtle body deformation capture, and high precision. - How were the two-character codes generated and applied to the suit? - Does the system offer head support? - What methods are available for remapping the reconstructed deforming mesh to a 3D bone rig? - How does the system handle triangulation, calibration, and camera matching across all cameras?

[@willbe3043]

Exciting! I can't wait to see more development on this.

[@ankachen7468]

We will continue working on this! The next step will be using a printed suit to replace the handwritten one and adding the hands and head.

[@caseyandtim]

@@ankachen7468 There seem to be inexpensive, custom morph suit manufacturers around. I'm very excited about this project. The movement is very convincing! Do you have any plans to distill down to a network with fewer input cameras?

[@ankachen7468]

@@caseyandtim In our testing the method works with 2 or 4 cameras. The price of each camera is 2000$, solution with cheaper cameras are also possible. With fewer camera, you might only be able to capture one size of body, and less flesh deformation. But you can still capture the full join angles information, without too much sacrifice actually....

[@caseyandtim]

@@ankachen7468 thank you for the prompt reply. That’s great news. I look forward to testing it out.

[@douglasteixeiradeabreu]

Fantástico 😃👏👏

[@eduardolarrymarinsilva76]

Amazing work! This should make mocap sessions faster to setup. Three questions though: 1. Since the suit has to be tight fitting to track the body precisely, does that mean that it would significantly dampen secondary flesh dynamics (as opposed to a naked body or looser fitting clothes)? If so, how would one generalize this tech to regain those motions. If not, then I'm interested if this has been formally tested so I can read about it. 2. The suit obviously won't fit an obese actor or children. Is designing suits for all body shapes accounted for in the methodology? Assuming it is, is the model able to adapt to these sizes without human supervision? 3. Can the model be enhanced to track two or more people interacting? Another observation is that the requirement for 16 cameras doesn't seem optimized. And all cameras having the same altitude doesn't seem wise.

[@ankachen7468]

Thank you for your insightful comments! Let me answer your question briefly:

[@ankachen7468]

1. The only reason for using a tight-fit suit is to remove the wrinkles due to looseness, it does not need to be super tight otherwise it will impede the movement. Since our suit is quite thin, it does not apply very strong force on the body. In practice, we can observe quite an amount of flesh dynamics, including the shaking of fat tissues. We will make some experiments to test our abilities to capture those secondary flesh dynamics. 2. The suits shown in the video are bought from Amazon, with standard "M" "S" size. However we indeed developed an algorithm to generate a suit template based on the detailed body measurements of the capture subject. However, it will involve suit printing and sewing. 3. There are two ways to track multiple subjects: (1) using the same suit for everyone, and relying on tracking to differ the actors like the traditional mocap (2) a unique suit for everyone, our coding space can support up to 8 actors acting together without duplicating the code on the suit. Thank you for the advice on the camera setup, we will improve that!!

[@mhbstudio3155]

Very interesting job

[@WhiteDragon103]

I wonder how the suit would compare to drawing checkerboard patterns directly on the skin.

[@evenemptier1]

Great work, this is impressive! Have you tested using fiducial markers instead of a checkerboard? Something like AprilTag seems like it'd be a perfect fit here

[@gauravupasani9088]

What platform do researchers in the computer graphics industry use to demonstrate their new algorithms or models ? Like the visualization of the suit here or the viscosity simulation of the Stanford bunny ?

[@kartikshrivastava1500]

Writing a renderer from scratch for their purpose is the one I'm aware of.

[@weizhuo1842]

Can this capture system be applied to children? Pediatrician are long searching for a way to diagnose children with developmental delay in their balance/gross motor skills.

[@ankachen7468]

Yes，it can, the system has no requirement for capture subject, and we would love to cooperate to help children. You can email me [e-mail]

[@flowerflower1154]

I wanna get this suit for a party costume

[@ankachen7468]

The unpainted one can be bought on Amazon

[@flowerflower1154]

@@ankachen7468 Noooo I want painted :(