Abstract:Due to the scarcity of manually annotated data required for fine-grained video understanding, few-shot fine-grained (FS-FG) action recognition has gained significant attention, with the aim of classifying novel fine-grained action categories with only a few labeled instances. Despite the progress made in FS coarse-grained action recognition, current approaches encounter two challenges when dealing with the fine-grained action categories: the inability to capture subtle action details and the insufficiency of learning from limited data that exhibit high intra-class variance and inter-class similarity. To address these limitations, we propose M$^3$Net, a matching-based framework for FS-FG action recognition, which incorporates \textit{multi-view encoding}, \textit{multi-view matching}, and \textit{multi-view fusion} to facilitate embedding encoding, similarity matching, and decision making across multiple viewpoints. \textit{Multi-view encoding} captures rich contextual details from the intra-frame, intra-video, and intra-episode perspectives, generating customized higher-order embeddings for fine-grained data. \textit{Multi-view matching} integrates various matching functions enabling flexible relation modeling within limited samples to handle multi-scale spatio-temporal variations by leveraging the instance-specific, category-specific, and task-specific perspectives. \textit{Multi-view fusion} consists of matching-predictions fusion and matching-losses fusion over the above views, where the former promotes mutual complementarity and the latter enhances embedding generalizability by employing multi-task collaborative learning. Explainable visualizations and experimental results on three challenging benchmarks demonstrate the superiority of M$^3$Net in capturing fine-grained action details and achieving state-of-the-art performance for FS-FG action recognition.

Multi-cue based four-stream 3D ResNets for video-based action recognition

Human Action Recognition Based on Three-Stream Network with Frame Sequence Features

Learning SpatioTemporal and Motion Features in a Unified 2D Network for Action Recognition

Spatiotemporal Multimodal Learning With 3D CNNs for Video Action Recognition

Action Recognition By Learning Deep Multi-Granular Spatio-Temporal Video Representation

DC3D: A Video Action Recognition Network Based on Dense Connection

3-Stream Convolutional Networks for Video Action Recognition with Hybrid Motion Field

MSF-Net: A Multilevel Spatiotemporal Feature Fusion Network Combines Attention for Action Recognition

Action recognition method based on a novel keyframe extraction method and enhanced 3D convolutional neural network

Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization

M^3Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition

Multi-View Region Adaptive Multi-temporal DMM and RGB Action Recognition

Multi-modality Fusion Network for Action Recognition.

M$^3$Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition

Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition

3D Action Recognition Using Multi-Temporal Depth Motion Maps and Fisher Vector

Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences

Learning Comprehensive Motion Representation for Action Recognition

Spatio‐temporal attention modules in orientation‐magnitude‐response guided multi‐stream CNNs for human action recognition

Mixed Resolution Network with Hierarchical Motion Modeling for Efficient Action Recognition

Multi-head attention-based two-stream EfficientNet for action recognition