AdapterFL: Adaptive Heterogeneous Federated Learning for Resource-constrained Mobile Computing Systems

Ruixuan Liu,Ming Hu,Zeke Xia,Jun Xia,Pengyu Zhang,Yihao Huang,Yang Liu,Mingsong Chen

2023-11-23

Abstract:Federated Learning (FL) enables collaborative learning of large-scale distributed clients without data sharing. However, due to the disparity of computing resources among massive mobile computing devices, the performance of traditional homogeneous model-based Federated Learning (FL) is seriously limited. On the one hand, to achieve model training in all the diverse clients, mobile computing systems can only use small low-performance models for collaborative learning. On the other hand, devices with high computing resources cannot train a high-performance large model with their insufficient raw data. To address the resource-constrained problem in mobile computing systems, we present a novel heterogeneous FL approach named AdapterFL, which uses a model reassemble strategy to facilitate collaborative training of massive heterogeneous mobile devices adaptively. Specifically, we select multiple candidate heterogeneous models based on the computing performance of massive mobile devices and then divide each heterogeneous model into two partitions. By reassembling the partitions, we can generate models with varied sizes that are combined by the partial parameters of the large model with the partial parameters of the small model. Using these reassembled models for FL training, we can train the partial parameters of the large model using low-performance devices. In this way, we can alleviate performance degradation in large models due to resource constraints. The experimental results show that AdapterFL can achieve up to 12\% accuracy improvement compared to the state-of-the-art heterogeneous federated learning methods in resource-constrained scenarios.

Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively utilize heterogeneous devices for Federated Learning (FL) in resource-constrained mobile computing systems. Specifically, the paper focuses on the following challenges:

1. **Device Heterogeneity**: The computational capabilities of mobile computing devices vary greatly. Traditional homogeneous federated learning methods can only choose the lowest-performing device to determine the global model, which leads to underutilization of high-performance devices' resources, while low-performance devices cannot train high-precision large models.

2. **Data Limitation**: Each device has a limited amount of data, making it difficult to train high-precision models on a few high-performance devices.

3. **Non-Independent and Identically Distributed (Non-IID) Data**: The data collected by different devices is influenced by the environment and user preferences, leading to the "client drift" problem, which affects the inference accuracy of the global model.

To address these challenges, the paper proposes a new heterogeneous federated learning framework—AdapterFL. Through model partitioning and reassembly strategies, AdapterFL can generate multiple heterogeneous models that adapt to different device hardware resources, thereby improving the performance of federated learning in resource-constrained scenarios. Specifically, AdapterFL divides each prototype model into two parts: a feature extraction block and a device adaptation block. By recombining these blocks, multiple initial global models can be generated, which can be locally trained on different devices and ultimately aggregated to update the global model. Experimental results show that AdapterFL can achieve up to a 12% accuracy improvement compared to existing heterogeneous federated learning methods in resource-constrained scenarios.