Abstract:Marker-less 3D human motion capture from a single colour camera has seen significant progress. However, it is a very challenging and severely ill-posed problem. In consequence, even the most accurate state-of-the-art approaches have significant limitations. Purely kinematic formulations on the basis of individual joints or skeletons, and the frequent frame-wise reconstruction in state-of-the-art methods greatly limit 3D accuracy and temporal stability compared to multi-view or marker-based motion capture. Further, captured 3D poses are often physically incorrect and biomechanically implausible, or exhibit implausible environment interactions (floor penetration, foot skating, unnatural body leaning and strong shifting in depth), which is problematic for any use case in computer graphics. We, therefore, present PhysCap, the first algorithm for physically plausible, real-time and marker-less human 3D motion capture with a single colour camera at 25 fps. Our algorithm first captures 3D human poses purely kinematically. To this end, a CNN infers 2D and 3D joint positions, and subsequently, an inverse kinematics step finds space-time coherent joint angles and global 3D pose. Next, these kinematic reconstructions are used as constraints in a real-time physics-based pose optimiser that accounts for environment constraints (e.g., collision handling and floor placement), gravity, and biophysical plausibility of human postures. Our approach employs a combination of ground reaction force and residual force for plausible root control, and uses a trained neural network to detect foot contact events in images. Our method captures physically plausible and temporally stable global 3D human motion, without physically implausible postures, floor penetrations or foot skating, from video in real time and in general scenes. The video is available at <a class="link-external link-http" href="http://gvv.mpi-inf.mpg.de/projects/PhysCap" rel="external noopener nofollow">this http URL</a>

Contact and Human Dynamics from Monocular Video

D&D: Learning Human Dynamics from Dynamic Camera

Physics-based Human Motion Estimation and Synthesis from Videos

Efficient Human Motion Reconstruction from Monocular Videos with Physical Consistency Loss.

Optimal-State Dynamics Estimation for Physics-based Human Motion Capture from Videos

3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data

Deep Physics-aware Inference of Cloth Deformation for Monocular Human Performance Capture

Contact-aware Human Motion Forecasting

PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos

Physics-Guided Human Motion Capture with Pose Probability Modeling

Leveraging depth cameras and wearable pressure sensors for full-body kinematics and dynamics capture

Kinematics-based 3D Human-Object Interaction Reconstruction from Single View

Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video

4D Human Body Capture from Egocentric Video via 3D Scene Grounding

Kinematics-Guided Reinforcement Learning for Object-Aware 3D Ego-Pose Estimation

PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time

Deep Kinematics Analysis for Monocular 3D Human Pose Estimation

Contact-Aware Retargeting of Skinned Motion

Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency