Abstract:Non-iid data is prevalent in real-world federated learning problems. Data heterogeneity can come in different types in terms of distribution shifts. In this work, we are interested in the heterogeneity that comes from concept shifts, i.e., shifts in the prediction across clients. In particular, we consider multi-task learning, where we want the model to adapt to the task of the client. We propose a parameter-efficient framework to tackle this issue, where each client learns to mix between parameter-efficient adaptors according to its task. We use Low-Rank Adaptors (LoRAs) as the backbone and extend its concept to other types of layers. We call our framework Federated Low-Rank Adaptive Learning (FLoRAL). This framework is not an algorithm but rather a model parameterization for a multi-task learning objective, so it can work on top of any algorithm that optimizes this objective, which includes many algorithms from the literature. FLoRAL is memory-efficient, and clients are personalized with small states (e.g., one number per adaptor) as the adaptors themselves are federated. Hence, personalization is--in this sense--federated as well. Even though clients can personalize more freely by training an adaptor locally, we show that collaborative and efficient training of adaptors is possible and performs better. We also show that FLoRAL can outperform an ensemble of full models with optimal cluster assignment, which demonstrates the benefits of federated personalization and the robustness of FLoRAL to overfitting. We show promising experimental results on synthetic datasets, real-world federated multi-task problems such as MNIST, CIFAR-10, and CIFAR-100. We also provide a theoretical analysis of local SGD on a relaxed objective and discuss the effects of aggregation mismatch on convergence.

Client Adaptation improves Federated Learning with Simulated Non-IID Clients

Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraints

Emulating Full Client Participation: A Long-Term Client Selection Strategy for Federated Learning

Personalized Federated Learning with Adaptive Feature Extraction and Category Prediction in Non-IID Datasets

Tackling Mavericks in Federated Learning via Adaptive Client Selection Strategy

Adaptive Federated Learning with Auto-Tuned Clients

Client Selection in Federated Learning under Imperfections in Environment

An EMD-Based Adaptive Client Selection Algorithm for Federated Learning in Heterogeneous Data Scenarios

FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learning

Performance Enhancement in Federated Learning by Reducing Class Imbalance of Non-IID Data

Adaptive Client Clustering for Efficient Federated Learning over Non-IID and Imbalanced Data

Towards Client Driven Federated Learning

Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning

Pretraining Client Selection Algorithm Based on a Data Distribution Evaluation Model in Federated Learning

Client Selection for Federated Learning With Non-IID Data in Mobile Edge Computing

Collaborative and Efficient Personalization with Mixtures of Adaptors

Towards Instance-adaptive Inference for Federated Learning

Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning

Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability

Communication-Efficient Federated Learning with Accelerated Client Gradient

Utilizing Free Clients in Federated Learning for Focused Model Enhancement