Abstract:Estimating touch contact and pressure in egocentric vision is a central task for downstream applications in Augmented Reality, Virtual Reality, as well as many robotic applications, because it provides precise physical insights into hand-object interaction and object manipulation. However, existing contact pressure datasets lack egocentric views and hand poses, which are essential for accurate estimation during in-situ operation, both for AR/VR interaction and robotic manipulation. In this paper, we introduce EgoPressure,a novel dataset of touch contact and pressure interaction from an egocentric perspective, complemented with hand pose meshes and fine-grained pressure intensities for each contact. The hand poses in our dataset are optimized using our proposed multi-view sequence-based method that processes footage from our capture rig of 8 accurately calibrated RGBD cameras. EgoPressure comprises 5.0 hours of touch contact and pressure interaction from 21 participants captured by a moving egocentric camera and 7 stationary Kinect cameras, which provided RGB images and depth maps at 30 Hz. In addition, we provide baselines for estimating pressure with different modalities, which will enable future developments and benchmarking on the dataset. Overall, we demonstrate that pressure and hand poses are complementary, which supports our intention to better facilitate the physical understanding of hand-object interactions in AR/VR and robotics research.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address the issue of hand contact and pressure estimation in augmented reality (AR), virtual reality (VR), and robotic manipulation. Specifically, the paper proposes a new dataset, **EgoPressure**, for hand contact and pressure estimation from a first-person perspective (egocentric vision). #### Main Objectives: 1. **Provide a dataset of hand contact and pressure from a first-person perspective**: Existing datasets lack hand posture and pressure data from a first-person perspective, which is crucial for accurate estimation in practical applications. 2. **Include high-precision hand posture and pressure intensity**: The EgoPressure dataset not only provides hand posture information but also detailed pressure intensity data, which is important for physical understanding in AR/VR and robotics research. 3. **Develop a multi-view sequence optimization method**: To obtain accurate hand postures, the paper proposes a multi-view sequence-based method to optimize hand postures, thereby improving the quality of the dataset. 4. **Provide benchmarks**: The paper offers baseline pressure estimation under various modalities to facilitate future research improvements and developments. #### Specific Contributions: - **Dataset Scale**: EgoPressure includes 5.0 hours of contact and pressure interaction data, recorded by 21 participants using a mobile first-person camera and 7 fixed Kinect cameras. - **High-Quality Hand Posture and Pressure Data**: Hand postures are obtained through a multi-view optimization method, and precise pressure intensity data is provided. - **Multi-View Synchronization**: The dataset includes 8 synchronized RGBD cameras, ensuring data consistency and accuracy. - **Benchmarks**: Baseline pressure estimation under various modalities is provided to promote future research development. Through the above work, the paper aims to advance the understanding of hand interaction and physical perception capabilities in the fields of AR/VR and robotics.

EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision

PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images

PressureVision: Estimating Hand Pressure from a Single RGB Image

The Visual-Inertial- Dynamical Multirotor Dataset

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation

FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors

EgoPoser: Robust Real-Time Egocentric Pose Estimation from Sparse and Intermittent Observations Everywhere

AssemblyHands: Towards Egocentric Activity Understanding via 3D Hand Pose Estimation

Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects

3D Hand Pose Estimation in Everyday Egocentric Images

Visual Contact Pressure Estimation for Grippers in the Wild

MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors

EgoHumans: An Egocentric 3D Multi-Human Benchmark

NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction

In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition

Egocentric Gesture Recognition for Head-Mounted AR devices

POV-Surgery: A Dataset for Egocentric Hand and Tool Pose Estimation During Surgical Activities

Dataset of Tactile Signatures of the Human Right Hand in Twenty-One Activities of Daily Living Using a High Spatial Resolution Pressure Sensor

EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understanding

Ego4D: Around the World in 3,000 Hours of Egocentric Video