Abstract:We present Ego3DPose, a highly accurate binocular egocentric 3D pose reconstruction system. The binocular egocentric setup offers practicality and usefulness in various applications, however, it remains largely under-explored. It has been suffering from low pose estimation accuracy due to viewing distortion, severe self-occlusion, and limited field-of-view of the joints in egocentric 2D images. Here, we notice that two important 3D cues, stereo correspondences, and perspective, contained in the egocentric binocular input are neglected. Current methods heavily rely on 2D image features, implicitly learning 3D information, which introduces biases towards commonly observed motions and leads to low overall accuracy. We observe that they not only fail in challenging occlusion cases but also in estimating visible joint positions. To address these challenges, we propose two novel approaches. First, we design a two-path network architecture with a path that estimates pose per limb independently with its binocular heatmaps. Without full-body information provided, it alleviates bias toward trained full-body distribution. Second, we leverage the egocentric view of body limbs, which exhibits strong perspective variance (e.g., a significantly large-size hand when it is close to the camera). We propose a new perspective-aware representation using trigonometry, enabling the network to estimate the 3D orientation of limbs. Finally, we develop an end-to-end pose reconstruction network that synergizes both techniques. Our comprehensive evaluations demonstrate that Ego3DPose outperforms state-of-the-art models by a pose estimation error (i.e., MPJPE) reduction of 23.1% in the UnrealEgo dataset. Our qualitative results highlight the superiority of our approach across a range of scenarios and challenges.

EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understanding

EgoNCE++: Do Egocentric Video-Language Models Really Understand Hand-Object Interactions?

In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition

EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation

EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views

Egocentric Hand-object Interaction Detection and Application

Pattern4Ego: Learning Egocentric Video Representation Using Cross-video Activity Patterns

EgoHumans: An Egocentric 3D Multi-Human Benchmark

HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction

Ego3DT: Tracking Every 3D Object in Ego-centric Videos

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

Helping Hands: An Object-Aware Ego-Centric Video Recognition Model

Egocentric Human Activities Recognition With Multimodal Interaction Sensing

EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World

Ego3DPose: Capturing 3D Cues from Binocular Egocentric Views

Ego+X: an Egocentric Vision System for Global 3D Human Pose Estimation and Social Interaction Characterization

Object Aware Egocentric Online Action Detection

EgoPAT3Dv2: Predicting 3D Action Target from 2D Egocentric Vision for Human-Robot Interaction

Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects

Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives

Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos