Abstract:Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focused on this problem and proposed improved models, such as CN-CLIP and AltCLIP, to facilitate their applicability to Chinese and even other languages. Nevertheless, these models suffer from high latency and a large memory footprint in inference, which limits their further deployment on resource-constrained edge devices. In this work, we propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. During the first stage, lightweight image/text student models are designed to learn robust visual/multilingual textual feature representation ability from corresponding teacher models, respectively. Subsequently, the multilingual vision-language alignment stage enables effective alignment of visual and multilingual textual features to further improve the model's multilingual performance. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context and competitive performance in the Chinese context, even with less training data, when compared to existing models of similar parameter magnitude. The evaluation demonstrates the effectiveness of our designed training mechanism.

Mclip: Multilingual CLIP Via Cross-lingual Transfer.

A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation

LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models

EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling

How Much Can CLIP Benefit Vision-and-Language Tasks?

Contrastive Localized Language-Image Pre-Training

Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Jina CLIP: Your CLIP Model Is Also Your Text Retriever

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger.

DiffCLIP: Few-shot Language-driven Multimodal Classifier

MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization

CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling

CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment

CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance

Iclip: Bridging Image Classification and Contrastive Language-Image Pre-Training for Visual Recognition

SpaceCLIP: A Vision-Language Pretraining Framework With Spatial Reconstruction On Text

VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts