Abstract:This paper, presents a multi‐stream CNN‐based model that leverages various attention modules to analyze multimodal data obtained from video cameras and Kinect sensors. A novel motion encoding technique, orientation‐magnitude response maps, is introduced to effectively capture the historical motion of actions, representing long‐frame sequences as a single motion history image. This paper introduces a new descriptor called orientation‐magnitude response maps as a single 2D image to effectively explore motion patterns. Moreover, boosted multi‐stream CNN‐based model with various attention modules is designed for human action recognition. The model incorporates a convolutional self‐attention autoencoder to represent compressed and high‐level motion features. Sequential convolutional self‐attention modules are used to exploit the implicit relationships within motion patterns. Furthermore, 2D discrete wavelet transform is employed to decompose RGB frames into discriminative coefficients, providing supplementary spatial information related to the actors actions. A spatial attention block, implemented through the weighted inception module in a CNN‐based structure, is designed to weigh the multi‐scale neighbours of various image patches. Moreover, local and global body pose features are combined by extracting informative joints based on geometry features and joint trajectories in 3D space. To provide the importance of specific channels in pose descriptors, a multi‐scale channel attention module is proposed. For each data modality, a boosted CNN‐based model is designed, and the action predictions from different streams are seamlessly integrated. The effectiveness of the proposed model is evaluated across multiple datasets, including HMDB51, UTD‐MHAD, and MSR‐daily activity, showcasing its potential in the field of action recognition.

Spatio‐temporal attention modules in orientation‐magnitude‐response guided multi‐stream CNNs for human action recognition

Multi-View Region Adaptive Multi-temporal DMM and RGB Action Recognition

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Pose-conditioned Spatio-Temporal Attention for Human Action Recognition

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

Integrating Temporal and Spatial Attention for Video Action Recognition

Spatiotemporal Multimodal Learning With 3D CNNs for Video Action Recognition

Local-aware spatio-temporal attention network with multi-stage feature fusion for human action recognition

CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Channel attention and multi-scale graph neural networks for skeleton-based action recognition

CANet: Comprehensive Attention Network for video-based action recognition

Unified Spatio-Temporal Attention Models for Advanced Human Action Recognition & Detection

Multimodal Activity Recognition with Local Block CNN and Attention-Based Spatial Weighted CNN.

Multi-level Channel Attention Excitation Network for Human Action Recognition in Videos

Spatio-Temporal Attention Networks for Action Recognition and Detection

An Efficient Lightweight Spatio-temporal Attention Module for Action Recognition.

ACTION-Net: Multipath Excitation for Action Recognition

An End to End Framework with Adaptive Spatio-Temporal Attention Module for Human Action Recognition.

A Spatio-temporal Hybrid Network for Action Recognition

Multi-stream with Deep Convolutional Neural Networks for Human Action Recognition in Videos

Channel Attention Module for Efficient Action Recognition