Cooperative Multi-Model Training for Personalized Federated Learning over Heterogeneous Devices

Jian Xu,Shuo Wan,Yinchuan Li,Sichun Luo,Zhilin Chen,Yunfeng Shao,Zhitang Chen,Shao-Lun Huang,Linqi Song
DOI: https://doi.org/10.1109/jstsp.2024.3497660
IF: 7.695
2024-01-01
IEEE Journal of Selected Topics in Signal Processing
Abstract:Federated learning (FL) is an increasingly popular paradigm for protecting data privacy in machine learning systems. However, the data heterogeneity and high computation cost/latency are challenging barriers for employing FL in real-world applications with heterogeneous devices. In this paper, we propose a novel personalized FL framework named $\mathtt {CompFL}$ allowing cooperative training of models with varied structures to mitigate those issues. First, $\mathtt {CompFL}$ initializes a set of expert models in varied sizes and allows each client to choose one or multiple expert models for training according to their capacities. Second, $\mathtt {CompFL}$ combines the model decoupling strategy and local-global feature alignment to mitigate the adverse impact of label heterogeneity, where clients only share the feature extractor part for each model architecture. Third, to encourage mutual enhancement of various models, knowledge distillation in local training is further applied to improve the overall performance. To make our framework workable in real systems, we implement it in both centralized settings with server-coordinated parallel training, and decentralized settings with newly developed device-to-device training-forwarding schemes. Extensive experiments on benchmark datasets are conducted to verify the potential of our framework for personalized FL over heterogeneous devices.
What problem does this paper attempt to address?