Abstract:Fine-grained action recognition is one of the critical problems in video processing, which aims to recognize similar actions of subtle interactions between humans and objects. Inspired by the remarkable performance of the Transformer in natural language processing , Transformer has been applied to the fine-grained action recognition task. However, Transformer needs abundant training data and extra supervision to achieve comparable results with convolutional neural networks (CNNs). To address these issues, we propose a Convolutional Transformer Network (CTN), which integrates the merits of CNN ( e.g. , sharing weights, capturing low-level features in videos and locality) and the benefits of Transformer ( e.g ., dynamic attention and learning long-range dependencies). In this paper, we propose two modifications to the original Transformer: (i) We propose a video-to-tokens module that can extract tokens from extracted spatial-temporal features in videos by 3D convolutions instead of the direct token embedding from raw input video clips; (ii) We completely replace the linear mapping in multi-head self-attention layer with depth-wise convolutional mapping, which applies a depth-wise separable convolution operation on embedded token maps. With these two modifications, our approach can extract effective spatial-temporal features from videos and process the long sequences of tokens encountered in videos. Experimental results demonstrate that our proposed CTN can achieve state-of-the-art accuracy on two fine-grained action recognition datasets ( i.e ., Epic-Kitchens and Diving 48) with a small computational increase.

Sparse Transformer-Based Algorithm for Long-Short Temporal Association Action Recognition

Short-Term Action Recognition by 3D Convolutional Neural Network with Pixel-Wise Evidences

STAR: Sparse Transformer-based Action Recognition

Temporal Transformer Networks with Self-Supervision for Action Recognition.

Convolutional transformer network for fine-grained action recognition

DC3D: A Video Action Recognition Network Based on Dense Connection

Action Recognition by an Attention-Aware Temporal Weighted Convolutional Neural Network.

Bi-direction Hierarchical LSTM with Spatial-Temporal Attention for Action Recognition

Efficient Action Recognition with Introducing R(2+1)D Convolution to Improved Transformer

Integrating Temporal and Spatial Attention for Video Action Recognition

Temporal Sparse Feature Auto-Combination Deep Network for Video Action Recognition.

Sparse Feature Auto-Combination Deep Network for Video Action Recognition

Sparse Temporal Causal Convolution for Efficient Action Modeling

T-C3D: Temporal Convolutional 3D Network for Real-Time Action Recognition

Action recognition using attention-based spatio-temporal VLAD networks and adaptive video sequences optimization

Short-Term Action Learning for Video Action Recognition

Recurrent Temporal Sparse Autoencoder for Attention-Based Action Recognition.

MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection

Action Recognition in Videos with Spatio-Temporal Fusion 3D Convolutional Neural Networks

STCA: an action recognition network with spatio-temporal convolution and attention

An Attention Mechanism Based Convolutional LSTM Network for Video Action Recognition.