Abstract:Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions, though, it is important to minimize the number of sensors attached to the subject's body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers

Mpose: Environment- and Subject-Agnostic 3D Skeleton Posture Reconstruction Leveraging a Single Mmwave Device

EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs

3D Human Pose Estimation Based on Wearable IMUs and Multiple Camera Views

AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing

Ultra Inertial Poser: Scalable Motion Capture and Tracking from Sparse Inertial Sensors and Ultra-Wideband Ranging

Fusion Poser: 3D Human Pose Estimation Using Sparse IMUs and Head Trackers in Real Time

Reconstructing Human Pose from Inertial Measurements: A Generative Model-based Compressive Sensing Approach

Reconstructing 3D human pose and shape from a single image and sparse IMUs

Embodied Scene-aware Human Pose Estimation

Design Space Exploration on Efficient and Accurate Human Pose Estimation from Sparse IMU-Sensing

emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation

3D Human Pose Estimation via Intuitive Physics

Hybrid 3D Human Pose Estimation with Monocular Video and Sparse IMUs

HEMlets PoSh: Learning Part-Centric Heatmap Triplets for 3D Human Pose and Shape Estimation

IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds

Upper Body Pose Estimation Using Wearable Inertial Sensors and Multiplicative Kalman Filter

3D Human Pose Estimation with Single Image and Inertial Measurement Unit (IMU) Sequence

Towards Accurate Markerless Human Shape and Pose Estimation over Time