Abstract:Multi-label classification is an essential task utilized in a wide variety of real-world applications. Multi-label zero-shot learning is a method for classifying images into multiple unseen categories for which no training data is available, while in general zero-shot situations, the test set may include observed classes. The CLIP-Decoder is a novel method based on the state-of-the-art ML-Decoder attention-based head. We introduce multi-modal representation learning in CLIP-Decoder, utilizing the text encoder to extract text features and the image encoder for image feature extraction. Furthermore, we minimize semantic mismatch by aligning image and word embeddings in the same dimension and comparing their respective representations using a combined loss, which comprises classification loss and CLIP loss. This strategy outperforms other methods and we achieve cutting-edge results on zero-shot multilabel classification tasks using CLIP-Decoder. Our method achieves an absolute increase of 3.9% in performance compared to existing methods for zero-shot learning multi-label classification tasks. Additionally, in the generalized zero-shot learning multi-label classification task, our method shows an impressive increase of almost 2.3%.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the multi - label zero - shot learning (ZSL) task. Specifically, the author aims to develop a new method that can perform multi - label classification on images without training data, especially in zero - shot learning and generalized zero - shot learning (GZSL) scenarios. ### Main problems: 1. **Multi - label zero - shot classification**: In a multi - label classification task, each image can belong to multiple categories, and the goal of zero - shot learning is to classify unseen categories without any training data. 2. **Semantic mismatch problem**: Traditional methods have the problem of semantic mismatch when dealing with visual and text modalities, resulting in poor classification performance. 3. **Limitations of existing methods**: Although existing attention - based methods have advantages in multi - scale object recognition, they perform poorly in zero - shot learning scenarios and have high computational costs. ### Solutions: To address the above challenges, the author proposes the CLIP - Decoder method, whose main features include: - **Multi - modal representation learning**: Use the text encoder of the CLIP model to extract text features, use the image encoder to extract image features, and align the two to the same dimension through projection. - **Joint loss function**: Introduce a joint loss function that combines classification loss and CLIP loss to minimize semantic mismatch and improve classification accuracy. - **Bi - modal fusion**: Effectively fuse visual and text information through transformer decoder layers to enhance context understanding. ### Key contributions: 1. Design prompt templates for the NUS - WIDE dataset and verify the effects of different templates. 2. Propose a multi - modal prompting method to align visual and language representations using the CLIP model. 3. Design the CLIP - Decoder architecture to improve the performance of the transformer decoder layer through bi - modal fusion. 4. Achieve significant performance improvements in zero - shot and generalized zero - shot learning tasks, with absolute performance gains of 3.9% and 2.3% respectively. ### Summary: This paper solves the semantic mismatch problem in multi - label zero - shot classification by introducing the CLIP - Decoder method and improves classification performance, especially performing excellently in zero - shot and generalized zero - shot learning scenarios.

CLIP-Decoder : ZeroShot Multilabel Classification using Multimodal CLIP Aligned Representation

Deep Multiple Instance Learning for Zero-Shot Image Tagging

Online Zero-Shot Classification with CLIP

CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collections

DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training

Multimodal Multilabel Classification by CLIP

CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentation

Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP

PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts

Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss

Transductive Zero-Shot and Few-Shot CLIP

TagCLIP: Improving Discrimination Ability of Zero-Shot Semantic Segmentation

ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation

Zero-shot Learning with Regularized Cross-Modality Ranking.

Towards Alleviating Text-to-Image Retrieval Hallucination for CLIP in Zero-shot Learning

CICA: Content-Injected Contrastive Alignment for Zero-Shot Document Image Classification

Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP

Multimodal CLIP Inference for Meta-Few-Shot Image Classification

Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training