Abstract:Contrastive Language Image Pre-training (CLIP) has recently demonstrated success across various tasks due to superior feature representation empowered by image-text contrastive learning. However, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data. To handle this limitation, cluster discrimination has been proposed through iterative cluster assignment and classification. Nevertheless, most cluster discrimination approaches only define a single pseudo-label for each image, neglecting multi-label signals in the image. In this paper, we propose a novel Multi-Label Cluster Discrimination method named MLCD to enhance representation learning. In the clustering step, we first cluster the large-scale LAION-400M dataset into one million centers based on off-the-shelf embedding features. Considering that natural images frequently contain multiple visual objects or attributes, we select the multiple closest centers as auxiliary class labels. In the discrimination step, we design a novel multi-label classification loss, which elegantly separates losses from positive classes and negative classes, and alleviates ambiguity on decision boundary. We validate the proposed multi-label cluster discrimination method with experiments on different scales of models and pre-training datasets. Experimental results show that our method achieves state-of-the-art performance on multiple downstream tasks including linear probe, zero-shot classification, and image-text retrieval.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to address the shortcomings of existing visual representation learning methods in capturing the semantic structure of training data. Specifically: 1. **Limitations of Instance Discrimination**: - Existing instance - discrimination - based models, such as CLIP, perform well in cross - modal tasks, but they mainly rely on instance - level contrastive learning. This results in samples from two different instances always being regarded as negative sample pairs, regardless of their semantic similarity. This approach may wrongly push apart semantically similar samples, making it difficult to fully encode the semantic structure of the training data. 2. **Limitations of Cluster Discrimination**: - Cluster discrimination methods improve representation learning through iterative clustering and classification and can better capture the semantic structure. However, most cluster discrimination methods assign only one pseudo - label to each image, ignoring the multi - label signals (e.g., multiple objects or attributes) present in the image. This limits the model's understanding of complex image content. 3. **The Need for Multi - label Classification**: - Natural images usually contain multiple visual objects or attributes, so a method that can handle multi - label signals is required. Existing methods either ignore these multi - label signals or encounter computational efficiency and accuracy challenges when dealing with large - scale data. To solve these problems, the authors propose a new Multi - Label Cluster Discrimination (MLCD) method. The main contributions of MLCD include: - **Multi - label Clustering**: In the clustering step, use the pre - trained CLIP model to perform offline clustering on a large - scale dataset (such as LAION - 400M) and select multiple closest cluster centers for each image as auxiliary category labels. - **Multi - label Classification Loss**: Design a novel multi - label classification loss function that elegantly separates the losses of positive and negative classes and alleviates the fuzziness of the decision boundary by optimizing intra - class similarity and inter - class similarity. Through these improvements, MLCD can achieve better performance than existing methods on multiple downstream tasks (such as linear probes, zero - shot classification, and image - text retrieval). ### Formula Summary - **Instance Discrimination Loss**: \[ L_{ID}=-\log\frac{\exp(e'^T_i e_i)}{\sum_{j = 1}^k\exp(e'^T_j e_i)} \] - **Cluster Discrimination Loss**: \[ L_{CD}=-\log\frac{\exp(w^T_{y_i} e_i)}{\sum_{j = 1}^k\exp(w^T_j e_i)}=-\log\frac{\exp(s_i)}{\sum_{j = 1}^k\exp(s_j)}=\log\left(1+\sum_{j = 1, j\neq i}^k\exp(s_j - s_i)\right) \] - **Multi - label Classification Loss**: \[ L_{MLC}=\log\left(1+\sum_{j\in\Omega_n}\exp(s_j)\sum_{i\in\Omega_p}\exp(-s_i)\right) \] - **Multi - label Classification Disambiguation Loss**: \[ L_{MLCD}=\log\left(1+\sum_{i\in\Omega_p}\exp(-s_i)\right)+\log\left(1+\sum_{j\in\Omega'_n}\exp(s_j)\right) \] where $\Omega_p$ represents the set of positive class labels, $\Omega_n$ represents the set of negative class labels, and $\Omega'_n$ is a randomly sampled subset from $\Omega_n$.

Multi-label Cluster Discrimination for Visual Representation Learning

Mejigclu: more effective jigsaw clustering for unsupervised visual representation learning

Multimodal Multilabel Classification by CLIP

CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination

Iclip: Bridging Image Classification and Contrastive Language-Image Pre-Training for Visual Recognition

DiffCLIP: Few-shot Language-driven Multimodal Classifier

Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering

On the Generalization of Multi-modal Contrastive Learning

Dual Contrast-Driven Deep Multi-View Clustering

Iterative Uni-modal and Cross-modal Clustered Contrastive Learning for Image-text Retrieval

Dual-Level Cross-Modal Contrastive Clustering

Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning

Balanced Contrastive Learning for Long-Tailed Visual Recognition

Diverse and Tailored Image Generation for Zero-shot Multi-label Classification

Multimodal Contrastive Training for Visual Representation Learning

View-Driven Multi-View Clustering via Contrastive Double-Learning

TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Training

LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models

Fine-Grained Representation Learning via Multi-Level Contrastive Learning without Class Priors

Deep dual incomplete multi-view multi-label classification via label semantic-guided contrastive learning

Triplet attention and dual-pool contrastive learning for clinic-driven multi-label medical image classification