ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model

Yifan Chen,Xiaozhen Qiao,Zhe Sun,Xuelong Li

2024-08-21

Abstract:Contrastive Language-Image Pre-training (CLIP) models excel in integrating semantic information between images and text through contrastive learning techniques. It has achieved remarkable performance in various multimodal tasks. However, the deployment of large CLIP models is hindered in resource-limited environments, while smaller models frequently fail to meet the performance benchmarks required for practical applications. In this paper, we propose a novel approach, ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model, which aims to comprehensively distill the knowledge from a large teacher CLIP model into a smaller student model, ensuring comparable performance with significantly reduced parameters. ComKD-CLIP is composed of two key mechanisms: Image Feature Alignment (IFAlign) and Educational Attention (EduAttention). IFAlign makes the image features extracted by the student model closely match those extracted by the teacher model, enabling the student to learn teacher's knowledge of extracting image features. EduAttention explores the cross-relationships between text features extracted by the teacher model and image features extracted by the student model, enabling the student model to learn how the teacher model integrates text-image features. In addition, ComKD-CLIP can refine the knowledge distilled from IFAlign and EduAttention by leveraging the text-image feature fusion results of the teacher model, ensuring the student model accurately absorbs the teacher's knowledge. Extensive experiments conducted on 11 datasets have demonstrated the superiority of the proposed method.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the challenges faced when deploying large CLIP models in resource-constrained environments and the performance inadequacies of small CLIP models in practical applications. Specifically, the paper proposes the ComKD-CLIP method, which aims to efficiently transfer the knowledge of a large teacher model to a small student model through comprehensive knowledge distillation techniques, ensuring that the student model maintains comparable performance despite a significant reduction in parameters.

The main contributions of the paper include:
1. Proposing the IFAlign module, which enables the student model to absorb the knowledge of image feature extraction from the teacher model.
2. Proposing the EduAttention module, which allows the student model to learn how the teacher model integrates text and image features.
3. Utilizing the feature fusion results of the teacher model to further optimize the distilled knowledge, ensuring the student model accurately absorbs the teacher model's knowledge.

Experimental results show that ComKD-CLIP outperforms various existing methods across multiple datasets, achieving the best performance in 8 different recognition datasets. This demonstrates the effectiveness and superiority of ComKD-CLIP in different tasks.