Abstract:Understanding how humans would behave during hand-object interaction is vital for applications in service robot manipulation and extended reality. To achieve this, some recent works have been proposed to simultaneously forecast hand trajectories and object affordances on human egocentric videos. The joint prediction serves as a comprehensive representation of future hand-object interactions in 2D space, indicating potential human motion and motivation. However, the existing approaches mostly adopt the autoregressive paradigm for unidirectional prediction, which lacks mutual constraints within the holistic future sequence, and accumulates errors along the time axis. Meanwhile, these works basically overlook the effect of camera egomotion on first-person view predictions. To address these limitations, we propose a novel diffusion-based interaction prediction method, namely Diff-IP2D, to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. We transform the sequential 2D images into latent feature space and design a denoising diffusion model to predict future latent interaction features conditioned on past ones. Motion features are further integrated into the conditional denoising process to enable Diff-IP2D aware of the camera wearer's dynamics for more accurate interaction prediction. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol. This highlights the efficacy of leveraging a generative paradigm for 2D hand-object interaction prediction. The code of Diff-IP2D will be released at <a class="link-external link-https" href="https://github.com/IRMVLab/Diff-IP2D" rel="external noopener nofollow">this https URL</a>.

Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction

Learning Object Affordance with Contact and Grasp Generation

S$^2$Contact: Graph-based Network for 3D Hand-Object Contact Estimation with Semi-Supervised Learning

Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images

ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation

Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing

Unsupervised Learning of Hierarchical Models for Hand-Object Interactions

Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos

Interaction and control with the auxiliary of hand gesture

Understanding Human Hands in Contact at Internet Scale

Dynamic Modeling of Hand-Object Interactions via Tactile Sensing

Learning a Contact Potential Field for Modeling the Hand-Object Interaction

A Method of Hand Gesture Segmentation and Tracking with Appearance Based on Probability Model

Joint Hand-Object Pose Estimation with Differentiably-Learned Physical Contact Point Analysis

Constraining Dense Hand Surface Tracking with Elasticity

Predicting Hand-Object Interaction for Improved Haptic Feedback in Mixed Reality

HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video

Fast and Robust Hand Tracking Using Detection-Guided Optimization

Background Mixup Data Augmentation for Hand and Object-in-Contact Detection

CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction

Weakly Supervised Segmentation Guided Hand Pose Estimation During Interaction with Unknown Objects.