Abstract:Immersive VR telepresence ideally means being able to interact and communicate with digital avatars that are indistinguishable from and precisely reflect the behaviour of their real counterparts. The core technical challenge is two fold: Creating a digital double that faithfully reflects the real human and tracking the real human solely from egocentric sensing devices that are lightweight and have a low energy consumption, e.g. a single RGB camera. Up to date, no unified solution to this problem exists as recent works solely focus on egocentric motion capture, only model the head, or build avatars from multi-view captures. In this work, we, for the first time in literature, propose a person-specific egocentric telepresence approach, which jointly models the photoreal digital avatar while also driving it from a single egocentric video. We first present a character model that is animatible, i.e. can be solely driven by skeletal motion, while being capable of modeling geometry and appearance. Then, we introduce a personalized egocentric motion capture component, which recovers full-body motion from an egocentric video. Finally, we apply the recovered pose to our character model and perform a test-time mesh refinement such that the geometry faithfully projects onto the egocentric view. To validate our design choices, we propose a new and challenging benchmark, which provides paired egocentric and dense multi-view videos of real humans performing various motions. Our experiments demonstrate a clear step towards egocentric and photoreal telepresence as our method outperforms baselines as well as competing methods. For more details, code, and data, we refer to our project page.

A Cybernetic Avatar System to Embody Human Telepresence for Connectivity, Exploration, and Skill Transfer

Anywhere Is Possible: An Avatar Platform for Social Telepresence With Full Perception of Physical Interaction

Analysis and Perspectives on the ANA Avatar XPRIZE Competition

Immersive Commodity Telepresence with the AVATRINA Robot Avatar

NimbRo wins ANA Avatar XPRIZE Immersive Telepresence Competition: Human-Centric Evaluation and Lessons Learned

Team Northeastern's Approach to ANA XPRIZE Avatar Final Testing: A Holistic Approach to Telepresence and Lessons Learned

NimbRo Avatar: Interactive Immersive Telepresence with Force-Feedback Telemanipulation

iCub3 avatar system: Enabling remote fully immersive embodiment of humanoid robots

Bimanual Telemanipulation with Force and Haptic Feedback through an Anthropomorphic Avatar System

EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars

Attention-Based VR Facial Animation with Visual Mouth Camera Guidance for Immersive Telepresence Avatars

Development of the Lifelike Head Unit for a Humanoid Cybernetic Avatar 'Yui' and its Operation Interface

X-Avatar: Expressive Human Avatars

An Avatar Robot Overlaid with the 3D Human Model of a Remote Operator

Interactive Animation of Virtual Characters: Application to Virtual Kung-Fu Fighting

Amplifying robotics capacities with a human touch: An immersive low-latency panoramic remote system

Virtual Co-Embodiment: Evaluation of the Sense of Agency while Sharing the Control of a Virtual Body among Two Individuals

VR Facial Animation for Immersive Telepresence Avatars

RAM-Avatar: Real-time Photo-Realistic Avatar from Monocular Videos with Full-body Control

A Full Body Avatar-Based Telepresence System for Dissimilar Spaces