Abstract:Large-scale pre-trained models and graph neural networks have recently demonstrated remarkable success in few-shot video classification tasks. However, they generally suffer from two key limitations: i) the temporal relations between adjacent frames tends to be ambiguous due to the lack of explicit temporal modeling. ii) the absence of multi-modal semantic knowledge in query videos results in inaccurate prototypes construction and an inability to achieve multi-modal temporal alignment metrics. To address these issues, we develop a Text-guided Graph Temporal Modeling (TgGTM) method that consists of two crucial components: a text-guided feature refinement module and a learnable Query text-token contrastive objective. Specifically, the former leverages the Temporal masking layer to guide the model in learning temporal relationships between adjacent frames. Additionally, it utilizes multi-modal information to refine video prototypes for comprehensive few-shot video classification. The latter addresses the feature discrepancy between multi-modal support features and single-modal query features by aligning a learnable Query text-token with corresponding base class text descriptions. Extensive experiments on four commonly used benchmarks demonstrate the effectiveness of our proposed method, which achieves mean accuracies of 54.4%, 80.3%, 91.9%, and 96.2% for 5-way 1-shot classification on SSV2-Small, HMDB51, Kinetics, and UCF101, respectively. These results are superior compared to existing state-of-the-art methods. A detailed ablation showcases the importance of learning temporal relationships between adjacent frames and obtaining Query text-token. The source code and models will be publicly available at https://github.com/JiaMingZhong2621/TgGTM.

Learning Implicit Temporal Alignment for Few-shot Video Classification

Few-Shot Video Classification via Temporal Alignment

Temporal Alignment Prediction for Few-Shot Video Classification

Few-shot action recognition with implicit temporal alignment and pair similarity optimization

Revisiting the Spatial and Temporal Modeling for Few-shot Action Recognition

Implicit Temporal Modeling with Learnable Alignment for Video Recognition

Learning Dynamic Alignment Via Meta-filter for Few-shot Learning

Searching for Better Spatio-temporal Alignment in Few-Shot Action Recognition

Elastic Temporal Alignment for Few‐shot Action Recognition

Learning to Compare Relation: Semantic Alignment for Few-Shot Learning

Category Alignment Mechanism for Few-Shot Image Classification

BDLA: Bi-directional local alignment for few-shot learning

Temporal Aggregation with Context Focusing for Few-Shot Video Object Detection

Text-Guided Graph Temporal Modeling for Few-Shot Video Classification

On the Importance of Spatial Relations for Few-shot Action Recognition

A Closer Look at Few-Shot Video Classification: A New Baseline and Benchmark

Few-Shot Ensemble Learning for Video Classification with SlowFast Memory Networks

Supervised Contrastive Learning for Few-Shot Action Classification

Temporal Alignment Prediction for Supervised Representation Learning and Few-Shot Sequence Classification

TTAN: Two-Stage Temporal Alignment Network for Few-shot Action Recognition

Few-Shot Action Recognition with Hierarchical Matching and Contrastive Learning