FLoCoRA: Federated learning compression with low-rank adaptation

Lucas Grativol Ribeiro,Mathieu Leonardon,Guillaume Muller,Virginie Fresse,Matthieu Arzel

2024-06-20

Abstract:Low-Rank Adaptation (LoRA) methods have gained popularity in efficient parameter fine-tuning of models containing hundreds of billions of parameters. In this work, instead, we demonstrate the application of LoRA methods to train small-vision models in Federated Learning (FL) from scratch. We first propose an aggregation-agnostic method to integrate LoRA within FL, named FLoCoRA, showing that the method is capable of reducing communication costs by 4.8 times, while having less than 1% accuracy degradation, for a CIFAR-10 classification task with a ResNet-8. Next, we show that the same method can be extended with an affine quantization scheme, dividing the communication cost by 18.6 times, while comparing it with the standard method, with still less than 1% of accuracy loss, tested with on a ResNet-18 model. Our formulation represents a strong baseline for message size reduction, even when compared to conventional model compression works, while also reducing the training memory requirements due to the low-rank adaptation.

Machine Learning,Signal Processing

What problem does this paper attempt to address?

The paper aims to address the issues of communication and computational efficiency in Federated Learning (FL). Specifically, the authors propose a new method called FLoCoRA (Federated Learning Compression with Low-Rank Adaptation), which utilizes Low-Rank Adaptation (LoRA) technology to train small visual models. Compared to traditional federated learning, this method can significantly reduce communication costs while achieving this goal with less than a 1% drop in accuracy.

The main contributions of the paper include:
1. **Proposing the FLoCoRA framework**: Integrating LoRA technology into the federated learning framework to make model training more efficient. By training and transmitting only the LoRA adapter parameters instead of the entire model, communication overhead is greatly reduced.
2. **Studying the impact of hyperparameters**: Investigating the impact of LoRA hyperparameters on classification tasks and identifying the settings that achieve optimal performance under different configurations.
3. **Introducing a quantization scheme**: Combining affine quantization techniques to further compress communication data. Experiments show that even after quantization, accuracy remains at a high level.
4. **Experimental validation**: Conducting extensive experiments to validate the effectiveness of FLoCoRA, particularly its performance on ResNet-8 and ResNet-18 models, demonstrating its ability to maintain high accuracy while reducing communication costs.

In summary, FLoCoRA provides an effective method to reduce communication overhead in federated learning while maintaining model performance, offering new possibilities for federated learning deployment on edge devices.