Abstract:Extracting effective spatial-temporal information is significantly important for video-based action recognition. Recently 3D convolutional neural networks (3D CNNs) that could simultaneously encode spatial and temporal dynamics in videos have made considerable progress in action recognition. However, almost all existing 3D CNN-based methods recognize human actions only using RGB videos. The single modality may limit the performance capacity of 3D networks. In this paper, we extend 3D CNN to depth and pose data besides RGB data to evaluate its capacity for spatiotemporal multimodal learning for video action recognition. We propose a novel multimodal two-stream 3D network framework, which can exploit complementary multimodal information to improve the recognition performance. Specifically, we first construct two discriminative video representations under depth and pose data modalities respectively, referred as depth residual dynamic image sequence (DRDIS) and pose estimation map sequence (PEMS). DRDIS captures spatial-temporal evolution of actions in depth videos by progressively aggregating the local motion information. PEMS eliminates the interference of cluttered backgrounds and describes the spatial configuration of body parts intuitively. The multimodal two-stream 3D CNN deals with two separate data streams to learn spatiotemporal features from DRDIS and PEMS representations. Finally, the classification scores from two streams are fused for action recognition. We conduct extensive experiments on four challenging action recognition datasets. The experimental results verify the effectiveness and superiority of our proposed method.

Spatiotemporal Pyramid Pooling In 3d Convolutional Neural Networks For Action Recognition

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Temporal Pyramid Pooling-Based Convolutional Neural Network for Action Recognition

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Spatiotemporal Pyramid Network for Video Action Recognition

Spatiotemporal Interaction Residual Networks with Pseudo3D for Video Action Recognition.

Dynamic Spatio-Temporal Feature Learning via Graph Convolution in 3D Convolutional Networks

D3D: Dual 3-D Convolutional Network for Real-Time Action Recognition

Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition

Spatio-Temporal Attention Networks for Action Recognition and Detection

Short-Term Action Recognition by 3D Convolutional Neural Network with Pixel-Wise Evidences

STH: Spatio-Temporal Hybrid Convolution for Efficient Action Recognition

DC3D: A Video Action Recognition Network Based on Dense Connection

Two-Stream 3-D Convnet Fusion for Action Recognition in Videos with Arbitrary Size and Length

Spatiotemporal Multimodal Learning With 3D CNNs for Video Action Recognition

End-to-end Video-level Representation Learning for Action Recognition

Spatio-temporal Laplacian Pyramid Coding for Action Recognition.

3D Convolutional Neural Network for Action Recognition.

Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition

Empowering Efficient Spatio-Temporal Learning with a 3D CNN for Pose-Based Action Recognition

MULTI-DIRECTIONAL CONVOLUTION NETWORKS WITH SPATIAL-TEMPORAL FEATURE PYRAMID MODULE FOR ACTION RECOGNITION