Abstract:We propose a new approach for 3D reconstruction of dynamic indoor and outdoor scenes in everyday environments, leveraging only cameras worn by a user. This approach allows 3D reconstruction of experiences at any location and virtual tours from anywhere. The key innovation of the proposed ego-centric reconstruction system is to capture the wearer's body pose and facial expression from near-body views, e.g. cameras on the user's glasses, and to capture the surrounding environment using outward-facing views. The main challenge of the ego-centric reconstruction, however, is the poor coverage of the near-body views - that is, the user's body and face are observed from vantage points that are convenient for wear but inconvenient for capture. To overcome these challenges, we propose a parametric-model-based approach to user motion estimation. This approach utilizes convolutional neural networks (CNNs) for near-view body pose estimation, and we introduce a CNN-based approach for facial expression estimation that combines audio and video. For each time-point during capture, the intermediate model-based reconstructions from these systems are used to re-target a high-fidelity pre-scanned model of the user. We demonstrate that the proposed self-sufficient, head-worn capture system is capable of reconstructing the wearer's movements and their surrounding environment in both indoor and outdoor situations without any additional views. As a proof of concept, we show how the resulting 3D-plus-time reconstruction can be immersively experienced within a virtual reality system (e.g., the HTC Vive). We expect that the size of the proposed egocentric capture-and-reconstruction system will eventually be reduced to fit within future AR glasses, and will be widely useful for immersive 3D telepresence, virtual tours, and general use-anywhere 3D content creation.

Scalable 3D Facial Shape Motion Retrieval from Image Sequences Using a Map-Reduce Framework

Recurrent Volume-based 3D Feature Fusion for Real-time Multi-view Object Pose Estimation

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation.

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

Video Tracked Facial Expression Animation

Visual Odometry Based 3D-Reconstruction

Displaced Dynamic Expression Regression for Real-Time Facial Tracking and Animation

Automatic 3D Face Modeling from Video

EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting

3D Shape Regression for Real-Time Facial Animation

Robust Real-Time Face Tracking And Modeling From Video

Real-Time Facial Segmentation and Performance Capture from RGB Input

Shape of Motion: 4D Reconstruction from a Single Video

Towards Fully Mobile 3D Face, Body, and Environment Capture Using Only Head-worn Cameras

Coherent 3D Portrait Video Reconstruction via Triplane Fusion

Real-Time Accurate 3D Head Tracking and Pose Estimation with Consumer RGB-D Cameras

Coherent3D: Coherent 3D Portrait Video Reconstruction via Triplane Fusion

Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera

CAD‐score: A new contact area difference‐based function for evaluation of protein structural models

Simultaneous Facial Landmark Detection, Pose and Deformation Estimation under Facial Occlusion

3d Reconstruction Of Dynamic Scenes With Multiple Handheld Cameras