Abstract:Existing studies of training state-of-the-art Contrastive Language-Image Pretraining (CLIP) models on large-scale data involve hundreds of or even thousands of GPUs due to the requirement of a large batch size. However, such a large amount of resources is not accessible to most people. While advanced compositional optimization techniques for optimizing global contrastive losses have been demonstrated effective for removing the requirement of large batch size, their performance on large-scale data remains underexplored and not optimized. To bridge the gap, this paper explores several aspects of CLIP training with limited resources (e.g., up to tens of GPUs). First, we introduce FastCLIP, a general CLIP training framework built on advanced compositional optimization techniques while designed and optimized for the distributed setting. Our framework is equipped with an efficient gradient reduction strategy to reduce communication overhead. Second, to further boost training efficiency, we investigate three components of the framework from an optimization perspective: the schedule of the inner learning rate, the update rules of the temperature parameter and the model parameters, respectively. Experiments on different strategies for each component shed light on how to conduct CLIP training more efficiently. Finally, we benchmark the performance of FastCLIP and the state-of-the-art training baseline (OpenCLIP) on different compute scales up to 32 GPUs on 8 nodes, and three data scales ranging from 2.7 million, 9.1 million to 315 million image-text pairs to demonstrate the significant improvement of FastCLIP in the resource-limited setting. We release the code of FastCLIP at <a class="link-external link-https" href="https://github.com/Optimization-AI/fast_clip" rel="external noopener nofollow">this https URL</a> .

What problem does this paper attempt to address?

The paper attempts to address the problem of how to efficiently train large-scale contrastive language-image pre-training (CLIP) models under limited resources (e.g., only a few dozen GPUs). Existing CLIP model training methods typically require hundreds or even thousands of GPUs because they rely on large batch data to achieve satisfactory results. However, such a large amount of computing resources is inaccessible to most researchers and small companies. Although some advanced combinatorial optimization techniques have been developed to reduce the need for large batch data, their performance on large-scale data has not been fully explored and optimized. Therefore, this paper aims to accelerate the training of CLIP models through a series of optimization techniques and enable them to be trained with limited computing resources. Specifically, the paper proposes the following points: 1. **FastCLIP Framework**: This is a general CLIP training framework based on advanced combinatorial optimization techniques, specifically designed and optimized for distributed settings. The framework adopts an efficient gradient reduction strategy to reduce communication overhead. 2. **Optimization Strategies**: To further improve training efficiency, the paper investigates three components of the framework from an optimization perspective: the scheduling of internal learning rates, the update rules for the temperature parameter, and the update rules for model parameters. 3. **Experimental Validation**: The paper evaluates the performance of FastCLIP on datasets and computing resources of different scales and compares it with existing state-of-the-art training baselines (such as OpenCLIP), demonstrating significant improvements of FastCLIP in resource-constrained environments. Through these methods, the paper hopes to achieve efficient and high-performance CLIP model training under limited resources.

FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources

Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and Training Strategies

Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers

Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

Improving CLIP Training with Language Rewrites

ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model

DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training

MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

FedCLIP: Fast Generalization and Personalization for CLIP in Federated Learning

A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation

GrowCLIP: Data-aware Automatic Model Growing for Large-scale Contrastive Language-Image Pre-training

ProtoCLIP: Prototypical Contrastive Language Image Pretraining

Scaling Language-Image Pre-training via Masking

CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Models

GraphCLIP: Enhancing Transferability in Graph Foundation Models for Text-Attributed Graphs

EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling

Long-CLIP: Unlocking the Long-Text Capability of CLIP

Enhancing Few-Shot CLIP With Semantic-Aware Fine-Tuning

DiffCLIP: Few-shot Language-driven Multimodal Classifier

Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement