Abstract:Significant challenges currently prohibit expressive interaction in virtual reality (VR). Occlusions introduced by head-mounted displays (HMDs) make existing facial tracking techniques intractable, and even state-of-the-art techniques used for real-time facial tracking in unconstrained environments fail to capture subtle details of the user's facial expressions that are essential for compelling speech animation. We introduce a novel system for HMD users to control a digital avatar in real-time while producing plausible speech animation and emotional expressions. Using a monocular camera attached to an HMD, we record multiple subjects performing various facial expressions and speaking several phonetically-balanced sentences. These images are used with artist-generated animation data corresponding to these sequences to train a convolutional neural network (CNN) to regress images of a user's mouth region to the parameters that control a digital avatar. To make training this system more tractable, we use audio-based alignment techniques to map images of multiple users making the same utterance to the corresponding animation parameters. We demonstrate that this approach is also feasible for tracking the expressions around the user's eye region with an internal infrared (IR) camera, thereby enabling full facial tracking. This system requires no user-specific calibration, uses easily obtainable consumer hardware, and produces high-quality animations of speech and emotional expressions. Finally, we demonstrate the quality of our system on a variety of subjects and evaluate its performance against state-of-the-art real-time facial tracking techniques.

Real-Time Human Motion Transfer System for Holographic Displays

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

High-Fidelity Neural Human Motion Transfer from Monocular Video

TeleHuman2

Real-time Animation and Motion Retargeting of Virtual Characters Based on Single RGB-D Camera.

Facial performance sensing head-mounted display

Ambulatory Real-Time Micro-Sensor Motion Capture

Human Motion Tracking with Less Constraint of Initial Posture from a Single RGB-D Sensor

HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations

3D Real Human Reconstruction Via Multiple Low-Cost Depth Cameras.

HMD$^2$: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device

A Real-Time 3d Human Body Tracking And Modeling System

Human Motion Transfer With 3D Constraints and Detail Enhancement

High-fidelity facial and speech animation for VR HMDs

Real-time 3D Human Motion Capture from Binocular Stereo Vision and Its Use on Human Body Animation

Neural3Points: Learning to Generate Physically Realistic Full-body Motion for Virtual Reality Users

Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors.

Human Motion Transfer from Poses in the Wild

Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras

Real-Time 3D Reconstruction Method for Holographic Telepresence

Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer-Grade VR Devices