CLIP-guided Federated Learning on Heterogeneous and Long-Tailed Data

Jiangming Shi,Shanshan Zheng,Xiangbo Yin,Yang Lu,Yuan Xie,Yanyun Qu

DOI: https://doi.org/10.48550/arXiv.2312.08648

2023-12-14

Abstract:Federated learning (FL) provides a decentralized machine learning paradigm where a server collaborates with a group of clients to learn a global model without accessing the clients' data. User heterogeneity is a significant challenge for FL, which together with the class-distribution imbalance further enhances the difficulty of FL. Great progress has been made in large vision-language models, such as Contrastive Language-Image Pre-training (CLIP), which paves a new way for image classification and object recognition. Inspired by the success of CLIP on few-shot and zero-shot learning, we use CLIP to optimize the federated learning between server and client models under its vision-language supervision. It is promising to mitigate the user heterogeneity and class-distribution balance due to the powerful cross-modality representation and rich open-vocabulary prior knowledge. In this paper, we propose the CLIP-guided FL (CLIP2FL) method on heterogeneous and long-tailed data. In CLIP2FL, the knowledge of the off-the-shelf CLIP model is transferred to the client-server models, and a bridge is built between the client and server. Specifically, for client-side learning, knowledge distillation is conducted between client models and CLIP to improve the ability of client-side feature representation. For server-side learning, in order to mitigate the heterogeneity and class-distribution imbalance, we generate federated features to retrain the server model. A prototype contrastive learning with the supervision of the text encoder of CLIP is introduced to generate federated features depending on the client-side gradients, and they are used to retrain a balanced server classifier.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the challenges of handling heterogeneous data and long-tailed distribution data in Federated Learning (FL). Specifically, the main issues faced in Federated Learning are:

1. **Data Heterogeneity**: The data from participating clients are independently distributed and have inconsistent sample distributions, leading to a significant drop in model accuracy.
2. **Class Imbalance**: Real-world data often exhibit a long-tail phenomenon, where some classes have a large number of samples while others have few. This imbalance can cause client models to be biased towards head classes, thereby affecting the performance of the global model.

To tackle these challenges, the authors propose a Federated Learning method based on CLIP (Contrastive Language-Image Pre-training) called CLIP2FL, which leverages CLIP's cross-modal representation capabilities and rich open vocabulary prior knowledge to optimize model training on both the client and server sides. CLIP2FL addresses the aforementioned issues in the following ways:

1. **Client-Side Learning**: By transferring CLIP's knowledge to local client models through knowledge distillation, it enhances the feature representation capabilities of client models and reduces bias towards head classes.
2. **Server-Side Learning**: By generating federated features and retraining the server model's classifier. Utilizing the prototype features provided by CLIP's text encoder ensures that the generated federated features are distinctive and semantically supervised, thereby alleviating data heterogeneity and class imbalance issues.

Experimental results show that CLIP2FL achieves significant performance improvements on multiple benchmark datasets, especially in handling long-tailed distribution data.