Abstract:The canonical video action recognition methods usually label categories with numbers or one-hot vectors and train neural networks to classify a fixed set of predefined categories, thereby constraining their ability to recognise complex actions and transferable ability to unseen concepts. In contrast, cross-modal learning can improve the performance of individual modalities. Based on the facts that a better action recogniser can be built by reading the statements used to describe actions, we exploited the recent multimodal foundation model CLIP for action recognition. In this study, an effective Vision-Language action recognition adaptation was implemented based on few-shot examples spanning different modalities. We added semantic information to action categories by treating textual and visual label as training examples for action classifier construction rather than simply labelling them with numbers. Due to the different importance of words in text and video frames, simply averaging all sequential features may result in ignoring keywords or key video frames. To capture sequential and hierarchical representation, a weighted token-wise interaction mechanism was employed to exploit the pair-wise correlations adaptively. Extensive experiments with public datasets show that cross-modal action recognition learning helps for downstream action images classification, in other words, the proposed method can train better action classifiers by reading the sentences describing action itself. The method proposed in this study not only reaches good generalisation and zero-shot/few-shot transfer ability on Out of Distribution (OOD) test sets, but also performs lower computational complexity due to the lightweight interaction mechanism with 84.15% Top-1 accuracy on the Kinetics-400.

GBC: Guided Alignment and Adaptive Boosting CLIP Bridging Vision and Language for Robust Action Recognition

ActionCLIP: Adapting Language-Image Pretrained Models for Video Action Recognition.

M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition

ActionCLIP: A New Paradigm for Video Action Recognition

EPK-CLIP: External and Priori Knowledge CLIP for action recognition

B2C-AFM: Bi-Directional Co-Temporal and Cross-Spatial Attention Fusion Model for Human Action Recognition.

Adapting CLIP for Action Recognition via Dual Semantic Supervision and Temporal Prompt Reparameterization

Rethinking CLIP-based Video Learners in Cross-Domain Open-Vocabulary Action Recognition

Action Recognition Via Fine-Tuned CLIP Model and Temporal Transformer.

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger.

Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications

RCAT: Retentive CLIP Adapter Tuning for Improved Video Recognition

Iclip: Bridging Image Classification and Contrastive Language-Image Pre-Training for Visual Recognition

Action Recognition Using Completed Local Binary Patterns and Multiple-Class Boosting Classifier.

PromptLearner-CLIP: Contrastive Multi-Modal Action Representation Learning with Context Optimization

ActivityCLIP: Enhancing Group Activity Recognition by Mining Complementary Information from Text to Supplement Image Modality

Fine-grained Knowledge Graph-driven Video-Language Learning for Action Recognition

Category-Specific Prompts for Animal Action Recognition with Pretrained Vision-Language Models

Cross-modal learning with multi-modal model for video action recognition based on adaptive weight training

AlignCLIP: Enhancing Stable Representations in Vision-Language Pretraining Models through Attention and Prediction Alignment

CdCLR: Clip-Driven Contrastive Learning for Skeleton-Based Action Recognition