Abstract:The canonical approach to video action recognition dictates a neural network model to do a classic and standard 1-of-N majority vote task. They are trained to predict a fixed set of predefined categories, limiting their transferability on new datasets with unseen concepts. In this article, we provide a new perspective on action recognition by attaching importance to the semantic information of label texts rather than simply mapping them into numbers. Specifically, we model this task as a video-text matching problem within a multimodal learning framework, which strengthens the video representation with more semantic language supervision and enables our model to do zero-shot action recognition without any further labeled data or parameters' requirements. Moreover, to handle the deficiency of label texts and make use of tremendous web data, we propose a new paradigm based on this multimodal learning framework for action recognition, which we dub "pre-train, adapt and fine-tune." This paradigm first learns powerful representations from pre-training on a large amount of web image-text or video-text data. Then, it makes the action recognition task to act more like pre-training problems via adaptation engineering. Finally, it is fine-tuned end-to-end on target datasets to obtain strong performance. We give an instantiation of the new paradigm, ActionCLIP, which not only has superior and flexible zero-shot/few-shot transfer ability but also reaches a top performance on general action recognition task, achieving 83.8% top-1 accuracy on Kinetics-400 with a ViT-B/16 as the backbone. Code is available at https://github.com/sallymmx/ActionCLIP.git.

The THUMOS challenge on action recognition for videos “in the wild”☆

Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos

Adsc submission at thumos challenge 2015

Uts-cmu at thumos 2015

Human Action Recognition Using Deep Learning Methods.

CUHK&SIAT submission for THUMOS15 action recognition challenge

A Comprehensive Study of Deep Video Action Recognition

Annotation-Efficient Untrimmed Video Action Recognition

Online Robust Action Recognition Based on a Hierarchical Model

Action Recognition by Exploring Data Distribution and Feature Correlation

ActionCLIP: Adapting Language-Image Pretrained Models for Video Action Recognition.

Hypergraph-based Multi-View Action Recognition using Event Cameras

Action Recognition Using HOG Feature in Different Resolution Video Sequences

Action Machine: Rethinking Action Recognition in Trimmed Videos

Human Action Recognition Based on DMMs, HOGs and Contourlet Transform

Action recognition and detection by combining motion and appearance features

Temporal Action Detection by Joint Identification-Verification.

TinyVIRAT: Low-resolution Video Action Recognition

HMDB: A large video database for human motion recognition

Action Recognition In Rgb-D Egocentric Videos

Action Recognition with Actons