Abstract:Action recognition is a fundamental and challenging task in computer vision. In recent years, optical flow, as the auxiliary information of frames in a video, has been widely applied to action recognition because of its advantage of utilizing the motion information of video data. However, existing methods only fuse the score of classification probabilities of the two streams; they do not consider the interaction between the image frames and the optical flows. In addition, the other important challenges lie in capturing significant motion information to be able to recognize the action. To overcome these problems, an action recognition model based on a multi-view temporal attention mechanism is proposed in this paper. Specifically, global temporal attention pooling is first designed to fuse multiple frame image features, where more attention is given to discriminative frames. Second, considering the complementarity of the image frame and optical flow, feature-level multi-view fusion methods are proposed. Experiments on three widely used benchmark datasets on action recognition show that our method outperforms other existing state-of-the-art methods. In addition, the effectiveness of the proposed method is extensively demonstrated under different factors, such as the temporal attention pooling strategy, multi-view feature fusion and network architecture. The promising experimental results demonstrate that introducing the temporal attention layer and feature-level multi-view fusion methods is of great effectiveness and overcomes the shortcomings of classical two-stream networks to some extent. Specifically, the proposed method has the following advantages. First, the temporal attention layer can accurately capture key frames that are more conducive to recognizing actions. Second, two kinds of features from image frames and optical flows are combined to make full use of their complementarity. Finally, a variety of fusion methods are employed for feature-level fusion instead of straightforward score fusion.

Action Anticipation in First-Person Videos with Self-Attention Based Multi-Modal Network

Multi-Label Action Anticipation for Real-World Videos with Scene Understanding

From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation

Weakly Supervised Action Anticipation Without Object Annotations

Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videos

Self-Supervised Learning of Video Representation for Anticipating Actions in Early Stage

Multimodal Large Models Are Effective Action Anticipators

Delving into 3D Action Anticipation from Streaming Videos

Action recognition using attention-based spatio-temporal VLAD networks and adaptive video sequences optimization

An Egocentric Action Anticipation Framework via Fusing Intuition and Analysis

Learning to Anticipate Egocentric Actions by Imagination

Self-Regulated Learning for Egocentric Video Activity Anticipation

VS-TransGRU: A Novel Transformer-GRU-based Framework Enhanced by Visual-Semantic Fusion for Egocentric Action Anticipation

Action Recognition with a Multi-View Temporal Attention Network

Unified Spatio-Temporal Attention Networks for Action Recognition in Videos.

Multi-Head TrajectoryCNN: A New Multi-Task Framework for Action Prediction

Anticipation and next action forecasting in video: an end-to-end model with memory

Skeleton-Indexed Deep Multi-Modal Feature Learning for High Performance Human Action Recognition

Weakly-Supervised Dense Action Anticipation

A Jeap-BiLSTM Neural Network for Action Recognition

META: Motion Excitation with Temporal Attention for Compressed Video Action Recognition