Abstract:We propose a new approach for 3D reconstruction of dynamic indoor and outdoor scenes in everyday environments, leveraging only cameras worn by a user. This approach allows 3D reconstruction of experiences at any location and virtual tours from anywhere. The key innovation of the proposed ego-centric reconstruction system is to capture the wearer's body pose and facial expression from near-body views, e.g. cameras on the user's glasses, and to capture the surrounding environment using outward-facing views. The main challenge of the ego-centric reconstruction, however, is the poor coverage of the near-body views - that is, the user's body and face are observed from vantage points that are convenient for wear but inconvenient for capture. To overcome these challenges, we propose a parametric-model-based approach to user motion estimation. This approach utilizes convolutional neural networks (CNNs) for near-view body pose estimation, and we introduce a CNN-based approach for facial expression estimation that combines audio and video. For each time-point during capture, the intermediate model-based reconstructions from these systems are used to re-target a high-fidelity pre-scanned model of the user. We demonstrate that the proposed self-sufficient, head-worn capture system is capable of reconstructing the wearer's movements and their surrounding environment in both indoor and outdoor situations without any additional views. As a proof of concept, we show how the resulting 3D-plus-time reconstruction can be immersively experienced within a virtual reality system (e.g., the HTC Vive). We expect that the size of the proposed egocentric capture-and-reconstruction system will eventually be reduced to fit within future AR glasses, and will be widely useful for immersive 3D telepresence, virtual tours, and general use-anywhere 3D content creation.

Real-Time 3D Face Reconstruction and Gaze Tracking for Virtual Reality

3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction `

Real-time 3D Face-Eye Performance Capture of a Person Wearing VR Headset

Facial performance sensing head-mounted display

FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

REFA: Real-time Egocentric Facial Animations for Virtual Reality

Development of a Visual Reality Headset and Its Applications in 3D Interactive Bike Race Games

Foveated Light-Field Display and Real-Time Rendering for Virtual Reality

New Algorithm for 3d Facial Model Reconstruction and Its Application in Virtual Reality

Towards a Pipeline for Real-Time Visualization of Faces for VR-based Telepresence and Live Broadcasting Utilizing Neural Rendering

HeadsetOff: Enabling Photorealistic Video Conferencing on Economical VR Headsets

Facial Expressions Recognition Based on Convolutional Neural Networks for Mobile Virtual Reality.

Towards Fully Mobile 3D Face, Body, and Environment Capture Using Only Head-worn Cameras

1 . RELATED WORK 1 . 1 3 D Reconstruction

A General Method of Realistic Avatar Modeling and Driving for Head-Mounted Display Users

3D Gaze Tracking for Studying Collaborative Interactions in Mixed-Reality Environments

Real-time 3D Human Reconstruction and Rendering System from a Single RGB Camera

A New Algorithm for 3d Facial Model Reconstruction and Its Application in Vr

Real-Time 3D Eye Performance Reconstruction for RGBD Cameras

High-fidelity facial and speech animation for VR HMDs

A Robust and Real-Time Full 3D Reconstruction Method Based on Multiple Kinect