Abstract:Human pose estimation is a critical component in autonomous driving and parking, enhancing safety by predicting human actions. Traditional frame-based cameras and videos are commonly applied, yet, they become less reliable in scenarios under high dynamic range or heavy motion blur. In contrast, event cameras offer a robust solution for navigating these challenging contexts. Predominant methodologies incorporate event cameras into learning frameworks by accumulating events into event frames. However, such methods tend to marginalize the intrinsic asynchronous and high temporal resolution characteristics of events. This disregard leads to a loss in essential temporal dimension data, crucial for safety-critical tasks associated with dynamic human activities. To address this issue and to unlock the 3D potential of event information, we introduce two 3D event representations: the Rasterized Event Point Cloud (RasEPC) and the Decoupled Event Voxel (DEV). The RasEPC collates events within concise temporal slices at identical positions, preserving 3D attributes with statistical cues and markedly mitigating memory and computational demands. Meanwhile, the DEV representation discretizes events into voxels and projects them across three orthogonal planes, utilizing decoupled event attention to retrieve 3D cues from the 2D planes. Furthermore, we develop and release EV-3DPW, a synthetic event-based dataset crafted to facilitate training and quantitative analysis in outdoor scenes. On the public real-world DHP19 dataset, our event point cloud technique excels in real-time mobile predictions, while the decoupled event voxel method achieves the highest accuracy. Experiments reveal our proposed 3D representation methods' superior generalization capacities against traditional RGB images and event frame techniques. Our code and dataset are available at https://github.com/MasterHow/EventPointPose.

Ego+X: an Egocentric Vision System for Global 3D Human Pose Estimation and Social Interaction Characterization

xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera

PoseConvGRU: A Monocular Approach for Visual Ego-motion Estimation by Learning

SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera

EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere

Ego-Body Pose Estimation via Ego-Head Pose Estimation

Scene-aware Egocentric 3D Human Pose Estimation

Rethinking Human Pose Estimation for Autonomous Driving with 3D Event Representations.

EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors

EgoFish3D: Egocentric 3D Pose Estimation from a Fisheye Camera Via Self-Supervised Learning

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

EgoPoseFormer: A Simple Baseline for Stereo Egocentric 3D Human Pose Estimation

EgoHumans: An Egocentric 3D Multi-Human Benchmark

Ego3DPose: Capturing 3D Cues from Binocular Egocentric Views

3D Human Pose Perception from Egocentric Stereo Videos

EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting

Social EgoMesh Estimation

EgoHDM: An Online Egocentric-Inertial Human Motion Capture, Localization, and Dense Mapping System

EgoAvatar: Egocentric View-Driven and Photorealistic Full-body Avatars

EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views

Estimating Body and Hand Motion in an Ego-sensed World