Abstract:Few-shot action recognition is a challenging problem aimed at learning a model capable of adapting to recognize new categories using only a few labeled videos. Recently, some works use attention mechanisms to focus on relevant regions to obtain discriminative representations. Despite the significant progress, these methods still cannot attain outstanding performance due to insufficient examples and a scarcity of additional supplementary information. In this paper, we propose a novel Semantic-guided Spatio-temporal Attention (SGSTA) approach for few-shot action recognition. The main idea of SGSTA is to exploit the semantic information contained in the text embedding of labels to guide attention to more accurately capture the rich spatio-temporal context in videos when visual content is insufficient. Specifically, SGSTA comprises two essential components: a visual-text alignment module and a semantic-guided spatio-temporal attention module. The former is used to align visual features and text embeddings to eliminate semantic gaps between them. The latter is further divided into spatial attention and temporal attention. Firstly, a semantic-guided spatial attention is applied on the frame feature map to focus on semantically relevant spatial regions. Then, a semantic-guided temporal attention is used to encode the semantically enhanced temporal context with a temporal Transformer. Finally, use the spatio-temporally contextual representation obtained to learn relationship matching between support and query sequences. In this way, SGSTA can fully utilize rich semantic priors in label embeddings to improve class-specific discriminability and achieve accurate few-shot recognition. Comprehensive experiments on four challenging benchmarks demonstrate that the proposed SGSTA is effective and achieves competitive performance over existing state-of-the-art methods under various settings.

Self-Supervised Few-Shot Time-Series Segmentation for Activity Recognition

Revisiting the Spatial and Temporal Modeling for Few-shot Action Recognition

Self-Supervised Tuning for Few-Shot Segmentation

Mining Latent Classes for Few-shot Segmentation

Iterative Few-shot Semantic Segmentation from Image Label Text

An efficient framework for few-shot skeleton-based temporal action segmentation

Semantic-guided spatio-temporal attention for few-shot action recognition

Self-Support Few-Shot Semantic Segmentation

Timestamp-supervised Wearable-based Activity Segmentation and Recognition with Contrastive Learning and Order-Preserving Optimal Transport

Learning self-target knowledge for few-shot segmentation

Blessing Few-Shot Segmentation Via Semi-Supervised Learning with Noisy Support Images

A Self-Distillation Embedded Supervised Affinity Attention Model for Few-Shot Segmentation

Adaptive FSS: A Novel Few-Shot Segmentation Framework Via Prototype Enhancement

Learning Prototype from Unlabeled Regions for Few-shot Segmentation

A Self-Supervised Few-Shot Semantic Segmentation Method Based on Multi-Task Learning and Dense Attention Computation

Semi-Supervised Contrastive Learning for Few-Shot Segmentation of Remote Sensing Images

A Multi-Task Deep Learning Approach for Sensor-based Human Activity Recognition and Segmentation

Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training

Not Just Learning from Others but Relying on Yourself: A New Perspective on Few-Shot Segmentation in Remote Sensing

Semi-Supervised Few-Shot Segmentation with Noisy Support Images