Abstract:Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions, though, it is important to minimize the number of sensors attached to the subject's body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

PresSim: An End-to-end Framework for Dynamic Ground Pressure Profile Generation from Monocular Videos Using Physics-based 3D Simulation

PressureTransferNet: Human Attribute Guided Dynamic Ground Pressure Profile Transfer using 3D simulated Pressure Maps

From Kinematics To Dynamics: Estimating Center of Pressure and Base of Support from Video Frames of Human Motion

Bodies at Rest: 3D Human Pose and Shape Estimation from a Pressure Image using Synthetic Data

Learning Dynamics from Kinematics: Estimating 2D Foot Pressure Maps from Video Frames

Text me the data: Generating Ground Pressure Sequence from Textual Descriptions for HAR

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Leveraging depth cameras and wearable pressure sensors for full-body kinematics and dynamics capture

Contact and Human Dynamics from Monocular Video

Leveraging Digital Perceptual Technologies for Remote Perception and Analysis of Human Biomechanical Processes: A Contactless Approach for Workload and Joint Force Assessment

PhysMotion: Physics-Grounded Dynamics From a Single Image

Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation

Real-time Physics-based Motion Capture with Sparse Sensors

Physics-based Human Motion Estimation and Synthesis from Videos

PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos

3D Human Pose Estimation via Intuitive Physics

DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion

SimPoE: Simulated Character Control for 3D Human Pose Estimation

Leveraging RGB-Pressure for Whole-body Human-to-Humanoid Motion Imitation

PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation

QuestEnvSim: Environment-Aware Simulated Motion Tracking from Sparse Sensors