Abstract:Human motion capture from monocular videos has made significant progress in recent years. However, modern approaches often produce temporal artifacts, e.g. in form of jittery motion and struggle to achieve smooth and physically plausible motions. Explicitly integrating physics, in form of internal forces and exterior torques, helps alleviating these artifacts. Current state-of-the-art approaches make use of an automatic PD controller to predict torques and reaction forces in order to re-simulate the input kinematics, i.e. the joint angles of a predefined skeleton. However, due to imperfect physical models, these methods often require simplifying assumptions and extensive preprocessing of the input kinematics to achieve good performance. To this end, we propose a novel method to selectively incorporate the physics models with the kinematics observations in an online setting, inspired by a neural Kalman-filtering approach. We develop a control loop as a meta-PD controller to predict internal joint torques and external reaction forces, followed by a physics-based motion simulation. A recurrent neural network is introduced to realize a Kalman filter that attentively balances the kinematics input and simulated motion, resulting in an optimal-state dynamics prediction. We show that this filtering step is crucial to provide an online supervision that helps balancing the shortcoming of the respective input motions, thus being important for not only capturing accurate global motion trajectories but also producing physically plausible human poses. The proposed approach excels in the physics-based human pose estimation task and demonstrates the physical plausibility of the predictive dynamics, compared to state of the art. The code is available on <a class="link-external link-https" href="https://github.com/cuongle1206/OSDCap" rel="external noopener nofollow">this https URL</a>

Real-time Physics-based Motion Capture with Sparse Sensors

Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors

Ultra Inertial Poser: Scalable Motion Capture and Tracking from Sparse Inertial Sensors and Ultra-Wideband Ranging

Real-Time Human Motion Capture Based on Wearable Inertial Sensor Networks

Full-body Human Motion Reconstruction with Sparse Joint Tracking Using Flexible Sensors

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Leveraging depth cameras and wearable pressure sensors for full-body kinematics and dynamics capture

Accurate realtime full-body motion capture using a single depth camera

High-Accuracy Real-Time Whole-Body Human Motion Tracking Based on Constrained Nonlinear Kalman Filtering

Human Motion Capture Using Wireless Inertial Sensors

DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion

JointTracker: Real-time inertial kinematic chain tracking with joint position estimation

HybridFusion: Real-Time Performance Capture Using a Single Depth Sensor and Sparse IMUs

Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras

Optimal-State Dynamics Estimation for Physics-based Human Motion Capture from Videos

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

Synergetic Reconstruction from 2D Pose and 3D Motion for Wide-Space Multi-Person Video Motion Capture in the Wild

Fusion Poser: 3D Human Pose Estimation Using Sparse IMUs and Head Trackers in Real Time

Physical Non-inertial Poser (PNP): Modeling Non-inertial Effects in Sparse-inertial Human Motion Capture

Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generation