Abstract:In this paper, we focus on egocentric action anticipation from videos, which enables various applications, such as helping intelligent wearable assistants understand users' needs and enhance their capabilities in the interaction process. It requires intelligent systems to observe from the perspective of the first person and predict an action before it occurs. Owing to the uncertainty of future, it is insufficient to perform action anticipation relying on visual information especially when there exists salient visual difference between past and future. In order to alleviate this problem, which we call visual gap in this paper, we propose one novel Intuition-Analysis Integrated (IAI) framework inspired by psychological research, which mainly consists of three parts: Intuition-based Prediction Network (IPN), Analysis-based Prediction Network (APN) and Adaptive Fusion Network (AFN). To imitate the implicit intuitive thinking process, we model IPN as an encoder-decoder structure and introduce one procedural instruction learning strategy implemented by textual pre-training. On the other hand, we allow APN to process information under designed rules to imitate the explicit analytical thinking, which is divided into three steps: recognition, transitions and combination. Both the procedural instruction learning strategy in IPN and the transition step of APN are crucial to improving the anticipation performance via mitigating the visual gap problem. Considering the complementarity of intuition and analysis, AFN adopts attention fusion to adaptively integrate predictions from IPN and APN to produce the final anticipation results. We conduct experiments on the largest egocentric video dataset. Qualitative and quantitative evaluation results validate the effectiveness of our IAI framework, and demonstrate the advantage of bridging visual gap by utilizing multi-modal information, including both visual features of observed segments and sequential instructions of actions.

Egocentric Intention Object Prediction Based on a Human-Like Manner

Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics.

Gaze-based Human Intention Prediction in the Hybrid Foraging Search Task

MIDAS: Deep learning human action intention prediction from natural eye movement patterns

Egocentric Prediction of Action Target in 3D

Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intention

Digging Deeper into Egocentric Gaze Prediction

What Will I Do Next? The Intention from Motion Experiment

Intention Action Anticipation Model with Guide-Feedback Loop Mechanism

Intention-Conditioned Long-Term Human Egocentric Action Forecasting

Multimodal Sense-Informed Prediction of 3D Human Motions

Using Cross-Model EgoSupervision to Learn Cooperative Basketball Intention

Inferring Human Intentions from Predicted Action Probabilities

Understanding Atomic Hand-Object Interaction With Human Intention

Driver intention prediction based on multi-dimensional cross-modality information interaction

An Egocentric Action Anticipation Framework via Fusing Intuition and Analysis

Where and Why Are They Looking? Jointly Inferring Human Attention and Intentions in Complex Tasks

GIMO: Gaze-Informed Human Motion Prediction in Context.

Towards Intention Prediction for Handheld Robots: a Case of Simulated Block Copying

Learning and Inferring "Dark Matter" and Predicting Human Intents and Trajectories in Videos.