Abstract:Human motion capture from monocular videos has made significant progress in recent years. However, modern approaches often produce temporal artifacts, e.g. in form of jittery motion and struggle to achieve smooth and physically plausible motions. Explicitly integrating physics, in form of internal forces and exterior torques, helps alleviating these artifacts. Current state-of-the-art approaches make use of an automatic PD controller to predict torques and reaction forces in order to re-simulate the input kinematics, i.e. the joint angles of a predefined skeleton. However, due to imperfect physical models, these methods often require simplifying assumptions and extensive preprocessing of the input kinematics to achieve good performance. To this end, we propose a novel method to selectively incorporate the physics models with the kinematics observations in an online setting, inspired by a neural Kalman-filtering approach. We develop a control loop as a meta-PD controller to predict internal joint torques and external reaction forces, followed by a physics-based motion simulation. A recurrent neural network is introduced to realize a Kalman filter that attentively balances the kinematics input and simulated motion, resulting in an optimal-state dynamics prediction. We show that this filtering step is crucial to provide an online supervision that helps balancing the shortcoming of the respective input motions, thus being important for not only capturing accurate global motion trajectories but also producing physically plausible human poses. The proposed approach excels in the physics-based human pose estimation task and demonstrates the physical plausibility of the predictive dynamics, compared to state of the art. The code is available on <a class="link-external link-https" href="https://github.com/cuongle1206/OSDCap" rel="external noopener nofollow">this https URL</a>

Efficient Human Motion Reconstruction from Monocular Videos with Physical Consistency Loss.

Frame-by-frame Motion Retargeting with Self-Collision Avoidance from Diverse Human Demonstrations

Efficient 3D Recovery of Human Motion in Monocular Video

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

Physics-based Human Motion Estimation and Synthesis from Videos

Contact and Human Dynamics from Monocular Video

Towards robust 3d reconstruction of human motion from monocular video

Physical Based Motion Reconstruction from Videos Using Musculoskeletal Model

Physics-Guided Human Motion Capture with Pose Probability Modeling

A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions

MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency

Optimal-State Dynamics Estimation for Physics-based Human Motion Capture from Videos

An Adaptive Approach to Human Motion Tracking from Video

Automatic reconstruction of 3D human motion pose from uncalibrated monocular video sequences based on markerless human motion tracking

Dynamic Multi-Person Mesh Recovery From Uncalibrated Multi-View Cameras

Enhanced Spatio-Temporal Context for Temporally Consistent Robust 3D Human Motion Recovery from Monocular Videos

Shape of Motion: 4D Reconstruction from a Single Video

Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos

Simultaneous 3-D Human-Motion Tracking and Voxel Reconstruction