Abstract:Recent advancements in technology have brought forth new forms of interactive applications, such as the social metaverse, where end users interact with each other through their virtual avatars. In such applications, precise full-body tracking is essential for an immersive experience and a sense of embodiment with the virtual avatar. However, current motion capture systems are not easily accessible to end users due to their high cost, the requirement for special skills to operate them, or the discomfort associated with wearable devices. In this paper, we present MOVIN, the data-driven generative method for real-time motion capture with global tracking, using a single LiDAR sensor. Our autoregressive conditional variational autoencoder (CVAE) model learns the distribution of pose variations conditioned on the given 3D point cloud from <a class="link-external link-http" href="http://LiDAR.As" rel="external noopener nofollow">this http URL</a> a central factor for high-accuracy motion capture, we propose a novel feature encoder to learn the correlation between the historical 3D point cloud data and global, local pose features, resulting in effective learning of the pose prior. Global pose features include root translation, rotation, and foot contacts, while local features comprise joint positions and rotations. Subsequently, a pose generator takes into account the sampled latent variable along with the features from the previous frame to generate a plausible current pose. Our framework accurately predicts the performer's 3D global information and local joint details while effectively considering temporally coherent movements across frames. We demonstrate the effectiveness of our architecture through quantitative and qualitative evaluations, comparing it against state-of-the-art methods. Additionally, we implement a real-time application to showcase our method in real-world scenarios. MOVIN dataset is available at \url{<a class="link-external link-https" href="https://movin3d.github.io/movin_pg2023/" rel="external noopener nofollow">this https URL</a>}.

Real-Time Motion Capture Technology on a Live Theatrical Performance with Computer Generated Scenery

Interconnected virtual space and Theater. Practice as research on theater stage in the era of the network

Creating a Theatrical Experience on a Virtual Stage

Challenges of movement quality using motion capture in theatre

Innovative Applications of Natural Language Processing and Digital Media in Theatre and Performing Arts

On Stage Interactive Spatial AR for Drama Performance

Physical World to Virtual Reality – Motion Capture Technology in Dance Creation

An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing

Interactive Theatre Experience in Embodied + Wearable Mixed Reality Space

Performing Capture in Video Games and New Acting Practice in the Post-Digital Era

Research and realization of stage augmented reality system

Mixed-Reality Geometric Algebra Animation Methods for Gamified Intangible Heritage

Virtual Instrument Performances (VIP): A Comprehensive Review

Greek Traditional Dances 3D Motion Capturing and a Proposed Method for Identification Through Rhythm Pattern Analyses (Terpsichore Project)

Three Axis Kinematics Study for Motion Capture Using Augmented Reality

MOVIN: Real-time Motion Capture using a Single LiDAR

Integrating performer into a real-time augmented reality performance spatially by using a multi-sensory prop

Movement-based interactive dance performance.

The Transformation and Application of Virtual and Reality in Creative Teaching: A New Interpretation of the Triadic Ballet

A Study on the Innovation of Traditional Drama Performance Forms by Intelligent Media Based on AI-Assisted Analysis Framework

Virtual Character Animation Based on Affordable Motion Capture and Reconfigurable Tangible Interfaces