Abstract:Recent methods for action recognition always apply 3D Convolutional Neural Networks (CNNs) to extract spatiotemporal features and introduce optical flows to present motion features. Although achieving state-of-the-art performance, they are expensive in both time and space. In this paper, we propose to represent both two kinds of features in a unified 2D CNN without any 3D convolution or optical flows calculation. In particular, we first design a channel-wise spatiotemporal module to present the spatiotemporal features and a channel-wise motion module to encode feature-level motion features efficiently. Besides, we provide a distinctive illustration of the two modules from the frequency domain by interpreting them as advanced and learnable versions of frequency components. Second, we combine these two modules and an identity mapping path into one united block that can easily replace the original residual block in the ResNet architecture, forming a simple yet effective network dubbed STM network by introducing very limited extra computation cost and parameters. Third, we propose a novel Twins Training framework for action recognition by incorporating a correlation loss to optimize the inter-class and intra-class correlation and a siamese structure to fully stretch the training data. We extensively validate the proposed STM on both temporal-related datasets (i.e., Something-Something v1 & v2) and scene-related datasets (i.e., Kinetics-400, UCF-101, and HMDB-51). It achieves favorable results against state-of-the-art methods in all the datasets.

Space-time Neighborhood Based Hierarchical Descriptor for Action Recognition

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Reassessing Hierarchical Representation for Action Recognition in Still Images

Action Recognition Using Discriminative Spatio-Temporal Neighborhood Features

Online Robust Action Recognition Based on a Hierarchical Model

Beyond Spatial Pyramid Matching: Space-time Extended Descriptor for Action Recognition

Action Recognition Using Nonnegative Action Component Representation and Sparse Basis Selection

Research on Local Spatio-Temporal Features for Action Recognition

Extracting Hierarchical Spatial and Temporal Features for Human Action Recognition

Modeling Geometric-Temporal Context with Directional Pyramid Co-Occurrence for Action Recognition

Action Recognition Using 3D DAISY Descriptor

Human action recognition using late fusion and dimensionality reduction

Action Recognition Based on Multi-scale Oriented Neighborhood Features

Spatio-temporal Semantic Features for Human Action Recognition.

Action recognition using a hierarchy of feature groups

Robust Human Action Recognition Based on Spatio-Temporal Descriptors and Motion Temporal Templates

A Non-negative Low Rank and Sparse Model for Action Recognition.

Multi-Temporal Depth Motion Maps-Based Local Binary Patterns for 3-D Human Action Recognition

Action Recognition by Hierarchical Mid-level Action Elements

3D Action Recognition Using Multi-Temporal Depth Motion Maps and Fisher Vector

Human Action Recognition by Using Polyhedron Model-Based Spatio-Temporal Gradient Descriptor