Abstract:Recent two-stream deep Convolutional Neural Networks (ConvNets) have made significant progress in recognizing human actions in videos. Despite their success, methods extending the basic two-stream ConvNet have not systematically explored possible network architectures to further exploit spatiotemporal dynamics within video sequences. Further, such networks often use different baseline two-stream networks. Therefore, the differences and the distinguishing factors between various methods using Recurrent Neural Networks (RNN) or convolutional networks on temporally-constructed feature vectors (Temporal-ConvNet) are unclear. In this work, we first demonstrate a strong baseline two-stream ConvNet using ResNet-101. We use this baseline to thoroughly examine the use of both RNNs and Temporal-ConvNets for extracting spatiotemporal information. Building upon our experimental results, we then propose and investigate two different networks to further integrate spatiotemporal information: 1) temporal segment RNN and 2) Inception-style Temporal-ConvNet. We demonstrate that using both RNNs (using LSTMs) and Temporal-ConvNets on spatiotemporal feature matrices are able to exploit spatiotemporal dynamics to improve the overall performance. However, each of these methods require proper care to achieve state-of-the-art performance; for example, LSTMs require pre-segmented data or else they cannot fully exploit temporal information. Our analysis identifies specific limitations for each method that could form the basis of future work. Our experimental results on UCF101 and HMDB51 datasets achieve state-of-the-art performances, 94.1% and 69.0%, respectively, without requiring extensive temporal augmentation.

Temporal Inception Architecture for Action Recognition with Convolutional Neural Networks.

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Temporal Interaction and Excitation for Action Recognition

Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

TEINet: Towards an Efficient Architecture for Video Recognition.

T-C3D: Temporal Convolutional 3D Network for Real-Time Action Recognition

Short-Term Action Recognition by 3D Convolutional Neural Network with Pixel-Wise Evidences

Temporal Segment Networks for Action Recognition in Videos

Temporal Attentive Network for Action Recognition

TDN: Temporal Difference Networks for Efficient Action Recognition

Temporal Transformer Networks with Self-Supervision for Action Recognition.

Temporal Distinct Representation Learning for Action Recognition

Action recognition with temporal scale-invariant deep learning framework

NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition

DTA: Double LSTM with Temporal-Wise Attention Network for Action Recognition

STA-TSN: Spatial-Temporal Attention Temporal Segment Network for Action Recognition in Video.

TSI: Temporal Saliency Integration for Video Action Recognition

Temporal-Spatial Mapping for Action Recognition

A Two-Pathway Convolutional Neural Network with Temporal Pyramid Network for Action Recognition