Abstract:Large-scale pre-trained text-image models with dual-encoder architectures (such as CLIP) are typically adopted for various vision-language applications, including text-image retrieval. However,these models are still less practical on edge devices or for real-time situations, due to the substantial indexing and inference time and the large consumption of computational resources. Although knowledge distillation techniques have been widely utilized for uni-modal model compression, how to expand them to the situation when the numbers of modalities and teachers/students are doubled has been rarely studied. In this paper, we conduct comprehensive experiments on this topic and propose the fully-Connected knowledge interaction graph (Cona) technique for cross-modal pre-training distillation. Based on our findings, the resulting ConaCLIP achieves SOTA performances on the widely-used Flickr30K and MSCOCO benchmarks under the lightweight setting. An industry application of our method on an e-commercial platform further demonstrates the significant effectiveness of ConaCLIP.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is: Although existing large - scale pre - trained text - image models (such as CLIP) perform excellently in various vision - language tasks, they become less practical when applied in edge devices or real - time scenarios due to their large indexes, inference time and huge consumption of computing resources. Although knowledge distillation techniques have been widely used for single - modal model compression, there is still relatively little research on how to extend these techniques for the double - encoder structure (that is, the situation where the number of teacher and student models involving two modalities is doubled). Specifically, starting from large - scale pre - trained double - encoder models, the paper aims to focus on pre - training distillation to propose a series of smaller, faster and effective lightweight models, thereby improving the feasibility of these models in practical applications, especially on edge devices with limited computing resources or in dynamic indexing scenarios (such as private photo/message collections). To this end, the author introduced a brand - new pre - training distillation method - the fully - connected knowledge interaction graph (Cona), and explored it in cross - modal pre - training distillation. Through this method, the author hopes to significantly reduce the model size and inference time while maintaining performance. ### Main contributions 1. **Proposed a new pre - training distillation method**: Use the fully - connected knowledge interaction graph (Cona) for pre - training distillation of lightweight double - encoder models. 2. **Released multiple lightweight ConaCLIP models**: These models significantly outperform previous state - of - the - art models in the commonly used Flickr30K and MSCOCO benchmark tests. 3. **Provided real - world application cases**: Demonstrated the practical value of this method in industrial scenarios, especially the application effect on e - commerce platforms. ### Key points of the solution - **Fully - connected knowledge interaction graph (Cona)**: It includes not only traditional intra - modal teacher - student learning, but also intra - modal student - student learning, cross - modal teacher - student learning and cross - modal student - student learning. This comprehensive knowledge - interaction learning can enhance the robustness and effectiveness of the model. - **Multiple supervision strategies**: In order to better explore the potential of Cona, the author implemented and studied multiple supervision strategies, such as InfoNCE loss, feature distance (FD) loss, similarity distance (SD) loss, KL - divergence (KL - Div) loss, etc., and finally made each learning type contribute to the overall improvement. Through these innovations, ConaCLIP has achieved excellent performance improvement in the lightweight setting and demonstrated its high efficiency and competitiveness in practical applications.

ConaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval

CocaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval

CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination

ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model

Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP

CLIP-KD: An Empirical Study of CLIP Model Distillation

Enhancing CLIP Conceptual Embedding through Knowledge Distillation

Distilling Knowledge from Text-to-Image Generative Models Improves Visio-Linguistic Reasoning in CLIP

MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers

TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance

A Lightweight and Effective Multi-View Knowledge Distillation Framework for Text-Image Retrieval

EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling

InterCLIP: Adapting CLIP To Interactive Image Retrieval with Triplet Similarity

LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models

DiffCLIP: Few-shot Language-driven Multimodal Classifier

Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Improving CLIP Robustness with Knowledge Distillation and Self-Training

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger.

Mclip: Multilingual CLIP Via Cross-lingual Transfer.