Abstract:Egocentric early action prediction, which aims to recognize the on-going action in the video captured in the first-person view as early as possible before the action is fully executed, is a new yet challenging task due to the limited partial video input. Pioneer studies focused on solving this task with LSTMs as the backbone and simply compiling the observed video segment and unobserved video segment into a single vector, which hence suffer from two key limitations: lack the non-sequential relation modeling with the video snippet sequence and the correlation modeling between the observed and unobserved video segment. To address these two limitations, in this paper, we propose a novel multimodal TransfoRmer-based duAl aCtion prEdiction (mTRACE) model for the task of egocentric early action prediction, which consists of two key modules: the early (observed) segment action prediction module and the future (unobserved) segment action prediction module. Both modules take Transformer encoders as the backbone for encoding all the potential relations among the input video snippets, and involve several single-modal and multi-modal classifiers for comprehensive supervision. Different from previous work, each of the two modules outputs two multi-modal feature vectors: one for encoding the current input video segment, and the other one for predicting the missing video segment. For optimization, we design a two-stage training scheme, including the mutual enhancement stage and end-to-end aggregation stage. The former stage alternatively optimizes the two action prediction modules, where the correlation between the observed and unobserved video segment is modeled with a consistency regularizer, while the latter seamlessly aggregates the two modules to fully utilize the capacity of the two modules. Extensive experiments have demonstrated the superiority of our proposed model. We have released the codes and the corresponding parameters to benefit other researchers at https://trace729.wixsite.com/trace .

MiTFM: A Multi-View Information Fusion Method Based on Transformer for Next Activity Prediction of Business Processes.

Hidformer: Hierarchical Dual-Tower Transformer Using Multi-Scale Mergence for Long-Term Time Series Forecasting

A Spatio-Temporal Transformer Network for Human Motion Prediction in Human-Robot Collaboration

Attention Please: What Transformer Models Really Learn for Process Prediction

Generalizable Memory-driven Transformer for Multivariate Long Sequence Time-series Forecasting

Multi-scale Temporal Fusion Transformer for Incomplete Vehicle Trajectory Prediction

Online Activity Detection with Predicting and Classifying Transformers

MVSTT: A Multiview Spatial-Temporal Transformer Network for Traffic-Flow Forecasting

Multi-scale Interaction Transformer for Temporal Action Proposal Generation.

Transformer-based Multi-scale Feature Integration Network for Video Saliency Prediction

Multi-task prediction method of business process based on BERT and Transfer Learning

Multi-Task Time Series Forecasting With Shared Attention

Multi‐future Transformer: Learning Diverse Interaction Modes for Behaviour Prediction in Autonomous Driving

Multi-modal transform-based fusion model for new product sales forecasting

A new dual-channel Transformer-based Network for Remaining Useful Life Prediction

Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios

MgMViT: Multi-Granularity and Multi-Scale Vision Transformer for Efficient Action Recognition

FMViT: A multiple-frequency mixing Vision Transformer

Egocentric Early Action Prediction Via Multimodal Transformer-Based Dual Action Prediction

Multimodal Motion Prediction with Stacked Transformers