Abstract:We propose a new approach for 3D reconstruction of dynamic indoor and outdoor scenes in everyday environments, leveraging only cameras worn by a user. This approach allows 3D reconstruction of experiences at any location and virtual tours from anywhere. The key innovation of the proposed ego-centric reconstruction system is to capture the wearer's body pose and facial expression from near-body views, e.g. cameras on the user's glasses, and to capture the surrounding environment using outward-facing views. The main challenge of the ego-centric reconstruction, however, is the poor coverage of the near-body views - that is, the user's body and face are observed from vantage points that are convenient for wear but inconvenient for capture. To overcome these challenges, we propose a parametric-model-based approach to user motion estimation. This approach utilizes convolutional neural networks (CNNs) for near-view body pose estimation, and we introduce a CNN-based approach for facial expression estimation that combines audio and video. For each time-point during capture, the intermediate model-based reconstructions from these systems are used to re-target a high-fidelity pre-scanned model of the user. We demonstrate that the proposed self-sufficient, head-worn capture system is capable of reconstructing the wearer's movements and their surrounding environment in both indoor and outdoor situations without any additional views. As a proof of concept, we show how the resulting 3D-plus-time reconstruction can be immersively experienced within a virtual reality system (e.g., the HTC Vive). We expect that the size of the proposed egocentric capture-and-reconstruction system will eventually be reduced to fit within future AR glasses, and will be widely useful for immersive 3D telepresence, virtual tours, and general use-anywhere 3D content creation.

UNOC: Understanding Occlusion for Embodied Presence in Virtual Reality

2D Motion Detection Bounded Hand 3D Trajectory Tracking and Gesture Recognition under Complex Background

In-Hand 3D Object Reconstruction from a Monocular RGB Video

4D Human Body Capture from Egocentric Video via 3D Scene Grounding

Resolving hand‐object occlusion for mixed reality with joint deep learning and model optimization

Egocentric 6-DoF Tracking of Small Handheld Objects

Towards Unstructured Unlabeled Optical Mocap: A Video Helps!

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars

XR-MBT: Multi-modal Full Body Tracking for XR through Self-Supervision with Learned Depth Point Cloud Registration

Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video

Instance Tracking in 3D Scenes from Egocentric Videos

Multimodal Visual-haptic pose estimation in the presence of transient occlusion

Human Motion Tracking with Less Constraint of Initial Posture from a Single RGB-D Sensor

DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers

RobustFusion: Human Volumetric Capture with Data-Driven Visual Cues Using a RGBD Camera

Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion

A virtual reality platform for dynamic human-scene interaction

Tracking Occluded Objects And Recovering Incomplete Trajectories By Reasoning About Containment Relations And Human Actions

Towards Fully Mobile 3D Face, Body, and Environment Capture Using Only Head-worn Cameras

3D Human Pose Perception from Egocentric Stereo Videos