Abstract:Action recognition is essential for egocentric video understanding, allowing automatic and continuous monitoring of Activities of Daily Living (ADLs) without user effort. Existing literature focuses on 3D hand pose input, which requires computationally intensive depth estimation networks or wearing an uncomfortable depth sensor. In contrast, there has been insufficient research in understanding 2D hand pose for egocentric action recognition, despite the availability of user-friendly smart glasses in the market capable of capturing a single RGB image. Our study aims to fill this research gap by exploring the field of 2D hand pose estimation for egocentric action recognition, making two contributions. Firstly, we introduce two novel approaches for 2D hand pose estimation, namely EffHandNet for single-hand estimation and EffHandEgoNet, tailored for an egocentric perspective, capturing interactions between hands and objects. Both methods outperform state-of-the-art models on H2O and FPHA public benchmarks. Secondly, we present a robust action recognition architecture from 2D hand and object poses. This method incorporates EffHandEgoNet, and a transformer-based action recognition method. Evaluated on H2O and FPHA datasets, our architecture has a faster inference time and achieves an accuracy of 91.32% and 94.43%, respectively, surpassing state of the art, including 3D-based methods. Our work demonstrates that using 2D skeletal data is a robust approach for egocentric action understanding. Extensive evaluation and ablation studies show the impact of the hand pose estimation approach, and how each input affects the overall performance.

Eyewatchme-3d Hand And Object Tracking For Inside Out Activity Analysis

3D Hand and Object Tracking for Inside out Activity Analysis

3D Hand Tracking With Head Mounted Gaze-Directed Camera

A Single-Camera Gaze Tracking System under Natural Light

Hand tracking based on the combination of 2D and 3D model in gaze-directed video

2D Motion Detection Bounded Hand 3D Trajectory Tracking and Gesture Recognition under Complex Background

Ukf-Based Visual Tracking With Eye-In-Hand Camera

Spatial Cognition from Egocentric Video: Out of Sight, Not Out of Mind

Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing

3D Hand Pose Estimation in Everyday Egocentric Images

3D hand tracking for human computer interaction

What You See is What You Grasp: User-Friendly Grasping Guided by Near-eye-tracking

Hand's 3D movement detection with one handheld camera.

In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition

HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos

Uncalibrated Dynamic Eye-in-hand Visual Tracking of Robots by Observing Unknown Static Objects

EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation

InterTracker: Discovering and Tracking General Objects Interacting with Hands in the Wild

Instance Tracking in 3D Scenes from Egocentric Videos

Egocentric Object Recognition Leveraging The 3d Shape Of The Grasping Hand

3D-Aware Instance Segmentation and Tracking in Egocentric Videos