Abstract:High-quality motion reconstruction that follows the user's movements can be achieved by high-end mocap systems with many sensors. However, obtaining such animation quality with fewer input devices is gaining popularity as it brings mocap closer to the general public. The main challenges include the loss of end-effector accuracy in learning-based approaches, or the lack of naturalness and smoothness in IK-based solutions. In addition, such systems are often finely tuned to a specific number of trackers and are highly sensitive to missing data e.g., in scenarios where a sensor is occluded or malfunctions. In response to these challenges, we introduce DragPoser, a novel deep-learning-based motion reconstruction system that accurately represents hard and dynamic on-the-fly constraints, attaining real-time high end-effectors position accuracy. This is achieved through a pose optimization process within a structured latent space. Our system requires only one-time training on a large human motion dataset, and then constraints can be dynamically defined as losses, while the pose is iteratively refined by computing the gradients of these losses within the latent space. To further enhance our approach, we incorporate a Temporal Predictor network, which employs a Transformer architecture to directly encode temporality within the latent space. This network ensures the pose optimization is confined to the manifold of valid poses and also leverages past pose data to predict temporally coherent poses. Results demonstrate that DragPoser surpasses both IK-based and the latest data-driven methods in achieving precise end-effector positioning, while it produces natural poses and temporally coherent motion. In addition, our system showcases robustness against on-the-fly constraint modifications, and exhibits exceptional adaptability to various input configurations and changes.

PE-DLS: a novel method for performing real-time full-body motion reconstruction in VR based on Vive trackers

Mpose: Environment- and Subject-Agnostic 3D Skeleton Posture Reconstruction Leveraging a Single Mmwave Device

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Real-time Full-Body Motion Reconstruction and Recognition for Off-the-shelf VR Devices

Leveraging depth cameras and wearable pressure sensors for full-body kinematics and dynamics capture

AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

Full-body pose reconstruction and correction in virtual reality for rehabilitation training

Motion Reconstruction from Sparse Accelerometer Data Using PLSR

HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations

A Survey of Full-Body Motion Reconstruction in Immersive Virtual Reality Applications

DragPoser: Motion Reconstruction from Variable Sparse Tracking Signals via Latent Space Optimization

DivaTrack: Diverse Bodies and Motions from Acceleration‐Enhanced Three‐Point Trackers

Human Motion Tracking with Less Constraint of Initial Posture from a Single RGB-D Sensor

DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers

Fusion Poser: 3D Human Pose Estimation Using Sparse IMUs and Head Trackers in Real Time

Synergetic Reconstruction from 2D Pose and 3D Motion for Wide-Space Multi-Person Video Motion Capture in the Wild

Stratified Avatar Generation from Sparse Observations

Leveraging Two Kinect Sensors for Accurate Full-Body Motion Capture

EM-POSE: 3D Human Pose Estimation from Sparse Electromagnetic Trackers

Accurate realtime full-body motion capture using a single depth camera