Abstract:The number of categories for action recognition is growing rapidly and it has become increasingly hard to label sufficient training data for learning conventional models for all categories. Instead of collecting ever more data and labelling them exhaustively for all categories, an attractive alternative approach is “zero-shot learning” (ZSL). To that end, in this study we construct a mapping between visual features and a semantic descriptor of each action category, allowing new categories to be recognised in the absence of any visual training data. Existing ZSL studies focus primarily on still images, and attribute-based semantic representations. In this work, we explore word-vectors as the shared semantic space to embed videos and category labels for ZSL action recognition. This is a more challenging problem than existing ZSL of still images and/or attributes, because the mapping between video space-time features of actions and the semantic space is more complex and harder to learn for the purpose of generalising over any cross-category domain shift. To solve this generalisation problem in ZSL action recognition, we investigate a series of synergistic strategies to improve upon the standard ZSL pipeline. Most of these strategies are transductive in nature which means access to testing data in the training phase. First, we enhance significantly the semantic space mapping by proposing manifold-regularized regression and data augmentation strategies. Second, we evaluate two existing post processing strategies (transductive self-training and hubness correction), and show that they are complementary. We evaluate extensively our model on a wide range of human action datasets including HMDB51, UCF101, Olympic Sports and event datasets including CCV and TRECVID MED 13. The results demonstrate that our approach achieves the state-of-the-art zero-shot action recognition performance with a simple and efficient pipeline, and without supervised annotation of attributes. Finally, we present in-depth analysis into why and when zero-shot works, including demonstrating the ability to predict cross-category transferability in advance.

Temporal–Semantic Aligning and Reasoning Transformer for Audio-Visual Zero-Shot Learning

Joint Learning of Attended Zero-Shot Features and Visual-Semantic Mapping.

Temporal and cross-modal attention for audio-visual zero-shot learning

Modality-Fusion Spiking Transformer Network for Audio-Visual Zero-Shot Learning

Transformer-Based Approach Via Contrastive Learning for Zero-Shot Detection.

Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning

Characterizing Hierarchical Semantic-Aware Parts with Transformers for Generalized Zero-Shot Learning

Multi-modal Generative Adversarial Network for Zero-Shot Learning

Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language

ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection

Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning

ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos

HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning

Multi-Modal Zero-Shot Sign Language Recognition

DSPformer: Discovering Semantic Parts with Token Growth and Clustering for Zero-Shot Learning

TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning

TransZero: Attribute-guided Transformer for Zero-Shot Learning

Deep Semantic-Visual Alignment for Zero-Shot Remote Sensing Image Scene Classification

Transductive Zero-Shot Action Recognition by Word-Vector Embedding

Audio-visual Generalized Zero-shot Learning the Easy Way

Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning