Abstract:Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focused on this problem and proposed improved models, such as CN-CLIP and AltCLIP, to facilitate their applicability to Chinese and even other languages. Nevertheless, these models suffer from high latency and a large memory footprint in inference, which limits their further deployment on resource-constrained edge devices. In this work, we propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. During the first stage, lightweight image/text student models are designed to learn robust visual/multilingual textual feature representation ability from corresponding teacher models, respectively. Subsequently, the multilingual vision-language alignment stage enables effective alignment of visual and multilingual textual features to further improve the model's multilingual performance. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context and competitive performance in the Chinese context, even with less training data, when compared to existing models of similar parameter magnitude. The evaluation demonstrates the effectiveness of our designed training mechanism.

What problem does this paper attempt to address?

The main problem addressed by this paper is the applicability of pre-trained visual language models (such as CLIP) in a multilingual environment, particularly in the context of Chinese and other languages. Despite existing work such as CN-CLIP and AltCLIP extending CLIP to multiple languages, these models suffer from high latency and large memory usage during inference, limiting their deployment on resource-constrained edge devices. To address this issue, the paper proposes a simple yet effective framework for compressing multilingual CLIP called DC-CLIP, which trains a lightweight bilingual visual language model suitable for both Chinese and English environments. The framework consists of two stages: visual language feature distillation and visual language feature alignment. In the first stage, a lightweight image/text student model is designed to learn robust visual/multilingual text feature representations from the corresponding teacher model. In the second stage, effective alignment of visual and multilingual text features is achieved through contrastive learning strategies, further improving the multilingual performance of the model. Experiments demonstrate that DC-CLIP performs well in zero-shot image classification tasks, particularly in English environments, and remains competitive in Chinese environments even with limited training data, compared to existing models with similar parameter sizes. This validates the effectiveness of the proposed training mechanism. The study provides a new approach for optimizing model size and computational requirements, enabling efficient deployment on resource-constrained devices, especially in the context of multilingual visual understanding tasks.

A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

Mclip: Multilingual CLIP Via Cross-lingual Transfer.

LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models

Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

How Much Can CLIP Benefit Vision-and-Language Tasks?

CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction

Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment

Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning

CLIP-Adapter: Better Vision-Language Models with Feature Adapters

EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling

PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining

SpaceCLIP: A Vision-Language Pretraining Framework With Spatial Reconstruction On Text

ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference

CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination

Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model

ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation

Diffusion Feedback Helps CLIP See Better

AlignCLIP: Enhancing Stable Representations in Vision-Language Pretraining Models through Attention and Prediction Alignment

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger.