Abstract:Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions, though, it is important to minimize the number of sensors attached to the subject's body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this article, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet toward the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

SPA: Sparse Photorealistic Animation Using a Single RGB-D Camera

Robust Keyframe-based Dense SLAM with an RGB-D Camera.

Video-based Characters

UnstructuredFusion: Realtime 4D Geometry and Texture Reconstruction Using Commercial RGBD Cameras.

Human Performance Capture Using Multiple Handheld Kinects

Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras

Outdoor Markerless Motion Capture with Sparse Handheld Video Cameras

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Real-Time Geometry, Albedo, and Motion Reconstruction Using a Single RGB-D Camera

3D real-time human reconstruction with a single RGBD camera

Motion Capture Research: 3D Human Pose Recovery Based on RGB Video Sequences

Novel View Synthesis of Dynamic Human with Sparse Cameras.

LiveCap: Real-time Human Performance Capture from Monocular Video

Total-Recon: Deformable Scene Reconstruction for Embodied View Synthesis

VividPose: Advancing Stable Video Diffusion for Realistic Human Image Animation

Performance Capture Of Interacting Characters With Handheld Kinect

Synthesizing Moving People with 3D Control

Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras

Learning Dynamic Textures for Neural Rendering of Human Actors

Video-Driven Neural Physically-Based Facial Asset for Production

Robust and Accurate 3D Self-Portraits in Seconds