Abstract:Understanding how humans would behave during hand-object interaction is vital for applications in service robot manipulation and extended reality. To achieve this, some recent works have been proposed to simultaneously forecast hand trajectories and object affordances on human egocentric videos. The joint prediction serves as a comprehensive representation of future hand-object interactions in 2D space, indicating potential human motion and motivation. However, the existing approaches mostly adopt the autoregressive paradigm for unidirectional prediction, which lacks mutual constraints within the holistic future sequence, and accumulates errors along the time axis. Meanwhile, these works basically overlook the effect of camera egomotion on first-person view predictions. To address these limitations, we propose a novel diffusion-based interaction prediction method, namely Diff-IP2D, to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. We transform the sequential 2D images into latent feature space and design a denoising diffusion model to predict future latent interaction features conditioned on past ones. Motion features are further integrated into the conditional denoising process to enable Diff-IP2D aware of the camera wearer's dynamics for more accurate interaction prediction. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol. This highlights the efficacy of leveraging a generative paradigm for 2D hand-object interaction prediction. The code of Diff-IP2D will be released at <a class="link-external link-https" href="https://github.com/IRMVLab/Diff-IP2D" rel="external noopener nofollow">this https URL</a>.

DAP: Diffusion-based Affordance Prediction for Multi-modality Storage

MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion

AffordDP: Generalizable Diffusion Policy with Transferable Affordance

Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement

Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos

DexGrasp-Diffusion: Diffusion-based Unified Functional Grasp Synthesis Pipeline for Multi-Dexterous Robotic Hands

DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulation

AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions

GraspDiff: Grasping Generation for Hand-Object Interaction With Multimodal Guided Diffusion

Language-Guided Object-Centric Diffusion Policy for Collision-Aware Robotic Manipulation

GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policy

DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignment

DR.CPO: Diversified and Realistic 3D Augmentation via Iterative Construction, Random Placement, and HPR Occlusion

SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation

DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation

APEX: Ambidextrous Dual-Arm Robotic Manipulation Using Collision-Free Generative Diffusion Models

DexGrasp-Diffusion: Diffusion-based Unified Functional Grasp Synthesis Method for Multi-Dexterous Robotic Hands

PartAfford: Part-level Affordance Discovery from 3D Objects

D3RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation

Differentiable Particles for General-Purpose Deformable Object Manipulation

Task-driven Perception and Manipulation for Constrained Placement of Unknown Objects