Abstract:This paper investigates a challenging problem of zero-shot learning in the multi-label scenario (MLZSL), wherein the model is trained to recognize multiple unseen classes within a sample (e.g., an image) based on seen classes and auxiliary knowledge, e.g., semantic information. Existing methods usually resort to analyzing the relationship of various seen classes residing in a sample from the dimension of spatial or semantic characteristics and transferring the learned model to unseen ones. However, they neglect the integrity of local and global features. Although the use of the attention structure will accurately locate local features, especially objects, it will significantly lose its integrity, and the relationship between classes will also be affected. Rough processing of global features will also directly affect comprehensiveness. This neglect will make the model lose its grasp of the main components of the image. Relying only on the local existence of seen classes during the inference stage introduces unavoidable bias. In this paper, we propose a novel and comprehensive visual-semantic framework for MLZSL, dubbed Epsilon, to fully make use of such properties and enable a more accurate and robust visual-semantic projection. In terms of spatial information, we achieve effective refinement by group aggregating image features into several semantic prompts. It can aggregate semantic information rather than class information, preserving the correlation between semantics. In terms of global semantics, we use global forward propagation to collect as much information as possible to ensure that semantics are not omitted. Experiments on large-scale MLZSL benchmark datasets NUS-Wide and Open-Images-v4 demonstrate that the proposed Epsilon outperforms other state-of-the-art methods with large margins.

MLTU: mixup long-tail unsupervised zero-shot image classification on vision-language models

Text-Guided Mixup Towards Long-Tailed Image Categorization

Meta-Transfer Networks for Zero-Shot Learning

Transductive Unbiased Embedding for Zero-Shot Learning

Learn More from Less: Generalized Zero-Shot Learning with Severely Limited Labeled Data

Zero-shot Classification with Unseen Prototype Learning

Multi-modal Generative Adversarial Network for Zero-Shot Learning

Enabling Small Models for Zero-Shot Classification through Model Label Learning

LLM meets Vision-Language Models for Zero-Shot One-Class Classification

Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models

Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions

Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image Recognition

LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections

DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning

Application of CLIP for Efficient Zero-Shot Learning

Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning

Global-local prompts guided image-text embedding, alignment and aggregation for multi-label zero-shot learning

Holistically Associated Transductive Zero-Shot Learning

Transductive Multi-class and Multi-label Zero-shot Learning

A Probabilistic Zero-Shot Learning Method Via Latent Nonnegative Prototype Synthesis of Unseen Classes.

The Neglected Tails in Vision-Language Models