Abstract:Recognizing actions in videos is not a trivial task because video is an information-intensive media and includes multiple modalities. Moreover, on each modality, an action may only appear at some spatial regions, or only part of the temporal video segments may contain the action. A valid question is how to locate the attended spatial areas and selective video segments for action recognition. In this paper, we devise a general attention neural cell, called AttCell, that estimates the attention probability not only at each spatial location but also for each video segment in a temporal sequence. With AttCell, a unified Spatio-Temporal Attention Networks (STAN) is proposed in the context of multiple modalities. Specifically, STAN extracts the feature map of one convolutional layer as the local descriptors on each modality and pools the extracted descriptors with the spatial attention measured by AttCell as a representation of each segment. Then, we concatenate the representation on each modality to seek a consensus on the temporal attention, a priori, to holistically fuse the combined representation of video segments to the video representation for recognition. Our model differs from conventional deep networks, which focus on the attention mechanism, because our temporal attention provides a principled and global guidance across different modalities and video segments. Extensive experiments are conducted on four public datasets; UCF101, CCV, THUMOS14, and Sports-1M; our STAN consistently achieves superior results over several state-of-the-art techniques. More remarkably, we validate and demonstrate the effectiveness of our proposal when capitalizing on the different number of modalities.

Towards Gradient-based Time-Series Explanations through a SpatioTemporal Attention Network

STAA: Spatio-Temporal Attention Attribution for Real-Time Interpreting Transformer-based Video Models

Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation

STES: A Spatiotemporal Explanation Supervision Framework

Exploring a Gradient-based Explainable AI Technique for Time-Series Data: A Case Study of Assessing Stroke Rehabilitation Exercises

Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning

Spatiotemporal Transformer Neural Network for Time-Series Forecasting

Spatio-Temporal Self-Attention Network for Video Saliency Prediction

Spatial-Temporal Attention Network for Open-Set Fine-Grained Image Recognition

Spatial-temporal Concept Based Explanation of 3D ConvNets.

ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition

Spatiotemporal Two-Stream LSTM Network for Unsupervised Video Summarization

Exploiting Informative Video Segments for Temporal Action Localization.

Spiking Transformer with Spatial-Temporal Attention

Explainability of Speech Recognition Transformers Via Gradient-Based Attention Visualization

Hybrid Attention Spatial-Temporal Network for Video Saliency Prediction

Video-based Spatio-Temporal Scene Graph Generation with Efficient Self-Supervision Tasks

Unified Spatio-Temporal Attention Networks for Action Recognition in Videos.

STAN: Spatio-Temporal Attention Network for Next Location Recommendation

Space or time for video classification transformers

Attention-Guided Spatial Transformer Networks for Fine-Grained Visual Recognition