Abstract:As a promising paradigm federated Learning (FL) is widely used in privacy-preserving machine learning, which allows distributed devices to collaboratively train a model while avoiding data transmission among clients. Despite its immense potential, the FL suffers from bottlenecks in training speed due to client heterogeneity, leading to escalated training latency and straggling server aggregation. To deal with this challenge, a novel split federated learning (SFL) framework that pairs clients with different computational resources is proposed, where clients are paired based on computing resources and communication rates among clients, meanwhile the neural network model is split into two parts at the logical level, and each client only computes the part assigned to it by using the SL to achieve forward inference and backward training. Moreover, to effectively deal with the client pairing problem, a heuristic greedy algorithm is proposed by reconstructing the optimization of training latency as a graph edge selection problem. Simulation results show the proposed method can significantly improve the FL training speed and achieve high performance both in independent identical distribution (IID) and Non-IID data distribution.
What problem does this paper attempt to address?
This paper attempts to address the issue of training speed bottlenecks in Federated Learning (FL) caused by client heterogeneity. Specifically, Federated Learning allows distributed devices to collaboratively train models without transmitting data, but differences in client computing resources can lead to increased training delays, thereby affecting overall training efficiency. To tackle this challenge, the paper proposes a novel framework based on pairing and split learning, called FedPairing.
### Main Issues:
1. **Client Heterogeneity**: Significant differences in computing resources and communication rates among clients lead to some clients becoming "stragglers" during training, which in turn affects the overall training speed.
2. **Training Delay**: Due to client heterogeneity, traditional federated learning methods experience high delays during training, especially on non-independent and identically distributed (Non-IID) datasets.
### Solutions:
1. **Client Pairing**: By pairing clients with different computing resources, the computing power of high-resource clients is used to share the computational tasks of low-resource clients, thereby reducing training delays.
2. **Model Splitting**: The neural network model is logically split into two parts, with each client only computing the part assigned to it. Forward inference and backward training are achieved through Split Learning (SL).
3. **Greedy Algorithm**: A heuristic algorithm based on a greedy strategy is proposed, which reconstructs the training delay optimization problem as a graph edge selection problem, effectively solving the client pairing issue and further improving training efficiency.
### Main Contributions:
1. **Proposed a new SFL framework, FedPairing**, which effectively manages the straggler phenomenon through client pairing and model splitting, enhancing data privacy protection in federated learning.
2. **Designed a greedy algorithm**, which optimizes the client pairing process by considering the communication rates and computing resources among clients, significantly improving the training speed of the federated learning system.
3. **Experimental results show** that the proposed method significantly improves training speed and performance on both independent and identically distributed (IID) and non-independent and identically distributed (Non-IID) datasets.
### Experimental Validation:
- **Convergence Performance**: Experimental results show that FedPairing achieves the highest model accuracy at the end of training, improving by 4.1%, 1.8%, and 10.8% compared to traditional federated learning (vanilla FL), traditional split learning (vanilla SL), and SplitFed, respectively.
- **Impact of Pairing Mechanism**: By comparing random pairing, location-based pairing, and computation resource-based pairing, experimental results indicate that FedPairing's greedy pairing strategy performs best in terms of training time, reducing time consumption by 61.8%, 78.7%, and 14.1% compared to other methods, respectively.
- **Training Time Cost**: Among different algorithms, FedPairing has the lowest average time cost per communication round, at 1553 seconds, while vanilla FL and SplitFed are 8716 seconds and 1798 seconds, respectively.
In summary, this paper effectively addresses the training speed bottleneck issue in federated learning caused by client heterogeneity by proposing a novel framework based on pairing and split learning, significantly improving training efficiency and model performance.