Abstract:Understanding how humans would behave during hand-object interaction is vital for applications in service robot manipulation and extended reality. To achieve this, some recent works have been proposed to simultaneously forecast hand trajectories and object affordances on human egocentric videos. The joint prediction serves as a comprehensive representation of future hand-object interactions in 2D space, indicating potential human motion and motivation. However, the existing approaches mostly adopt the autoregressive paradigm for unidirectional prediction, which lacks mutual constraints within the holistic future sequence, and accumulates errors along the time axis. Meanwhile, these works basically overlook the effect of camera egomotion on first-person view predictions. To address these limitations, we propose a novel diffusion-based interaction prediction method, namely Diff-IP2D, to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. We transform the sequential 2D images into latent feature space and design a denoising diffusion model to predict future latent interaction features conditioned on past ones. Motion features are further integrated into the conditional denoising process to enable Diff-IP2D aware of the camera wearer's dynamics for more accurate interaction prediction. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol. This highlights the efficacy of leveraging a generative paradigm for 2D hand-object interaction prediction. The code of Diff-IP2D will be released at <a class="link-external link-https" href="https://github.com/IRMVLab/Diff-IP2D" rel="external noopener nofollow">this https URL</a>.

FIction: 4D Future Interaction Prediction from Video

FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations

What Will I Do Next? The Intention from Motion Experiment

Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos

Action-guided 3D Human Motion Prediction.

Peeking Into the Future: Predicting Future Person Activities and Locations in Videos

Probabilistic Future Prediction for Video Scene Understanding

Graphing the Future: Activity and Next Active Object Prediction using Graph-based Activity Representations

T3VIP: Transformation-based 3D Video Prediction

Intention-Conditioned Long-Term Human Egocentric Action Forecasting

Forecasting Future Videos from Novel Views via Disentangled 3D Scene Representation

Social Behavior Prediction from First Person Videos

FIMP: Future Interaction Modeling for Multi-Agent Motion Prediction

Relational Action Forecasting

Predicting the Future from First Person (Egocentric) Vision: A Survey

EMAG: Ego-motion Aware and Generalizable 2D Hand Forecasting from Egocentric Videos

FutureDepth: Learning to Predict the Future Improves Video Depth Estimation

Predicting Long-horizon Futures by Conditioning on Geometry and Time

StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipation

Anticipating Human Intention for Full-Body Motion Prediction in Object Grasping and Placing Tasks

Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos