Abstract:Recent advancements in technology have brought forth new forms of interactive applications, such as the social metaverse, where end users interact with each other through their virtual avatars. In such applications, precise full-body tracking is essential for an immersive experience and a sense of embodiment with the virtual avatar. However, current motion capture systems are not easily accessible to end users due to their high cost, the requirement for special skills to operate them, or the discomfort associated with wearable devices. In this paper, we present MOVIN, the data-driven generative method for real-time motion capture with global tracking, using a single LiDAR sensor. Our autoregressive conditional variational autoencoder (CVAE) model learns the distribution of pose variations conditioned on the given 3D point cloud from <a class="link-external link-http" href="http://LiDAR.As" rel="external noopener nofollow">this http URL</a> a central factor for high-accuracy motion capture, we propose a novel feature encoder to learn the correlation between the historical 3D point cloud data and global, local pose features, resulting in effective learning of the pose prior. Global pose features include root translation, rotation, and foot contacts, while local features comprise joint positions and rotations. Subsequently, a pose generator takes into account the sampled latent variable along with the features from the previous frame to generate a plausible current pose. Our framework accurately predicts the performer's 3D global information and local joint details while effectively considering temporally coherent movements across frames. We demonstrate the effectiveness of our architecture through quantitative and qualitative evaluations, comparing it against state-of-the-art methods. Additionally, we implement a real-time application to showcase our method in real-world scenarios. MOVIN dataset is available at \url{<a class="link-external link-https" href="https://movin3d.github.io/movin_pg2023/" rel="external noopener nofollow">this https URL</a>}.

SIDQL: An Efficient Keyframe Extraction and Motion Reconstruction Framework in Motion Capture

Learning a Deep Motion Interpolation Network for Human Skeleton Animations

A Multiscale-Contour-based Interpolation Framework for Generating a Time-Varying Quasi-Dense Point Cloud Sequence.

Motion Parameters Measurement of User-Defined Key Points Using 3D Pose Estimation

A 3D Human Motion Refinement Method Based on Sparse Motion Bases Selection.

Local Optimized and Scalable Frame-to-model SLAM

Robust Keyframe-based Dense SLAM with an RGB-D Camera.

Performance-Driven Motion Retrieval and its Usability Evaluation

Motion Keyframe Interpolation for Any Human Skeleton via Temporally Consistent Point Cloud Sampling and Reconstruction

Robust Camera Motion Estimation in Video Sequences

Video Motion Capture by Silhouette Analysis and Pose Optimization

Markerless Shape and Motion Capture From Multiview Video Sequences

A group of novel approaches and a toolkit for motion capture data reusing

Solving Short-Term Relocalization Problems In Monocular Keyframe Visual SLAM Using Spatial And Semantic Data

SCAPE-based Human Performance Reconstruction

Shape of Motion: 4D Reconstruction from a Single Video

HybridFusion: Real-Time Performance Capture Using a Single Depth Sensor and Sparse IMUs

MOVIN: Real-time Motion Capture using a Single LiDAR

Simultaneous 3-D Human-Motion Tracking and Voxel Reconstruction

New multi-view human motion capture framework

Scalable 3D Facial Shape Motion Retrieval from Image Sequences Using a Map-Reduce Framework