Abstract:The Contrastive Language-Image Pre-training (CLIP) model achieves strong generalization by using a large number of text-image pairs for contrastive learning. However, when it is transferred to action recognition, the following two questions remain to be solved: 1) How to guide the model to focus more on human-body-related regions to better align actions and text, and 2) How to make the model strengthen itself in a targeted manner to deal with difficult-to-classify categories. To solve these problems, a Guided alignment and adaptive Boosting CLIP (GBC) is proposed, which employs visual prior knowledge and benefits from both feature and decision aggregation in a boosting manner. During early training, visual prior knowledge related to human body is adopted, which enables the model to better align human actions with category text to be robust to distribution shift. At the later stage of training, the CLIP encoder is frozen, and multiple downstream feature & decision aggregation modules are sequentially generated and trained. In such way, the model is able to boost the performance from different perspectives in the Boosting manner and at a linearly increasing cost. Moreover, a class-adaptive re-weighting strategy is proposed to make the model focus more on optimizing categories that are difficult to classify. The effectiveness of our model is validated on six action recognition datasets (Kinetics-600, Kinetics-400, Jester, HMDB-51, UCF-101, and Mini-Kinetics-200), including both fully supervised and zero-shot experiments. Our model achieves superior results compared to state-of-the-art methods on all datasets.

Action Recognition Using Completed Local Binary Patterns and Multiple-Class Boosting Classifier.

Action Recognition Using 3d Histograms of Texture and A Multi-Class Boosting Classifier

Multiclass AdaBoost ELM and Its Application in LBP Based Face Recognition

Action Recognition Based on Depth Motion Map and Hybrid Classifier

Action Recognition Using Multi-Scale Histograms of Oriented Gradients Based Depth Motion Trail Images

Multi-Temporal Depth Motion Maps-Based Local Binary Patterns for 3-D Human Action Recognition

Action Recognition from Depth Sequences Using Weighted Fusion of 2D and 3D Auto-Correlation of Gradients Features

3D Action Recognition Using Multi-Temporal Depth Motion Maps and Fisher Vector

Boosted Multi-Class Semi-Supervised Learning for Human Action Recognition

Combining Multi-scale Directed Depth Motion Maps and Log-Gabor Filters for Human Action Recognition

GBC: Guided Alignment and Adaptive Boosting CLIP Bridging Vision and Language for Robust Action Recognition

CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Action Recognition Using DMM-Gabor Descriptor and Temporal Segmentation

Human action recognition based on multi-feature fusion and hierarchical BP-AdaBoost algorithm

Action Recognition from Depth Maps Using Deep Convolutional Neural Networks

DMMs-Based Multiple Features Fusion for Human Action Recognition

Multi-View Region Adaptive Multi-temporal DMM and RGB Action Recognition

Gradient Local Auto-Correlations and Extreme Learning Machine for Depth-Based Activity Recognition.

Recognizing actions using depth motion maps-based histograms of oriented gradients

Human Action Recognition Using Multilevel Depth Motion Maps

Human Action Recognition Based on DMMs, HOGs and Contourlet Transform