Abstract:Understanding how humans would behave during hand-object interaction is vital for applications in service robot manipulation and extended reality. To achieve this, some recent works have been proposed to simultaneously forecast hand trajectories and object affordances on human egocentric videos. The joint prediction serves as a comprehensive representation of future hand-object interactions in 2D space, indicating potential human motion and motivation. However, the existing approaches mostly adopt the autoregressive paradigm for unidirectional prediction, which lacks mutual constraints within the holistic future sequence, and accumulates errors along the time axis. Meanwhile, these works basically overlook the effect of camera egomotion on first-person view predictions. To address these limitations, we propose a novel diffusion-based interaction prediction method, namely Diff-IP2D, to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. We transform the sequential 2D images into latent feature space and design a denoising diffusion model to predict future latent interaction features conditioned on past ones. Motion features are further integrated into the conditional denoising process to enable Diff-IP2D aware of the camera wearer's dynamics for more accurate interaction prediction. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol. This highlights the efficacy of leveraging a generative paradigm for 2D hand-object interaction prediction. The code of Diff-IP2D will be released at <a class="link-external link-https" href="https://github.com/IRMVLab/Diff-IP2D" rel="external noopener nofollow">this https URL</a>.

Implicit Contact Diffuser: Sequential Contact Reasoning with Latent Point Cloud Diffusion

Diffusion-Informed Probabilistic Contact Search for Multi-Finger Manipulation

Contact-Implicit Model Predictive Control for Dexterous In-hand Manipulation: A Long-Horizon and Robust Approach

A Contact Model based on Denoising Diffusion to Learn Variable Impedance Control for Contact-rich Manipulation

ManiDext: Hand-Object Manipulation Synthesis via Continuous Correspondence Embeddings and Residual-Guided Diffusion

DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation

Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos

ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulation

Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics

Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance

Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images

Admittance Visuomotor Policy Learning for General-Purpose Contact-Rich Manipulations

Diff-LfD: Contact-aware Model-based Learning from Visual Demonstration for Robotic Manipulation Via Differentiable Physics-based Simulation and Rendering.

Simultaneous Learning of Contact and Continuous Dynamics

Implicit Contact Dynamics Modeling With Explicit Inertia Matrix Representation for Real-Time, Model-Based Control in Physical Environment

Tactile Probabilistic Contact Dynamics Estimation of Unknown Objects

Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulation

Learning Diffusion Policies from Demonstrations For Compliant Contact-rich Manipulation

DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation

D-Cubed: Latent Diffusion Trajectory Optimisation for Dexterous Deformable Manipulation

NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Model