Abstract:Federated Learning (FL) enables collaborations among clients for train machine learning models while protecting their data privacy. Existing FL simulation platforms that are designed from the perspectives of traditional distributed training, suffer from laborious code migration between simulation and production, low efficiency, low GPU utility, low scalability with high hardware requirements and difficulty of simulating stateful clients. In this work, we firstly demystify the challenges and bottlenecks of simulating FL, and design a new FL system named as FedML \texttt{Parrot}. It improves the training efficiency, remarkably relaxes the requirements on the hardware, and supports efficient large-scale FL experiments with stateful clients by: (1) sequential training clients on devices; (2) decomposing original aggregation into local and global aggregation on devices and server respectively; (3) scheduling tasks to mitigate straggler problems and enhance computing utility; (4) distributed client state manager to support various FL algorithms. Besides, built upon our generic APIs and communication interfaces, users can seamlessly transform the simulation into the real-world deployment without modifying codes. We evaluate \texttt{Parrot} through extensive experiments for training diverse models on various FL datasets to demonstrate that \texttt{Parrot} can achieve simulating over 1000 clients (stateful or stateless) with flexible GPU devices setting ($4 \sim 32$) and high GPU utility, 1.2 $\sim$ 4 times faster than FedScale, and 10 $\sim$ 100 times memory saving than FedML. And we verify that \texttt{Parrot} works well with homogeneous and heterogeneous devices in three different clusters. Two FL algorithms with stateful clients and four algorithms with stateless clients are simulated to verify the wide adaptability of \texttt{Parrot} to different algorithms.

PipeSFL: A Fine-Grained Parallelization Framework for Split Federated Learning on Heterogeneous Clients

ParallelSFL: A Novel Split Federated Learning Framework Tackling Heterogeneity Issues

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Effectively Heterogeneous Federated Learning: A Pairing and Split Learning Based Approach

Split Federated Learning Over Heterogeneous Edge Devices: Algorithm and Optimization

SDPipe: A Semi-Decentralized Framework for Heterogeneity-aware Pipeline-parallel Training.

Convergence Analysis of Split Federated Learning on Heterogeneous Data

SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustments

MP-SL: Multihop Parallel Split Learning

HSFL: an Efficient Split Federated Learning Framework Via Hierarchical Organization.

No One Idles: Efficient Heterogeneous Federated Learning with Parallel Edge and Server Computation.

Have Your Cake and Eat It Too: Toward Efficient and Accurate Split Federated Learning

Efficient Semi-Supervised Federated Learning for Heterogeneous Participants

Split-FL: An Efficient Online Federated Learning Framework with Constrained Computation and Streaming Data

PFSL: Personalized & Fair Split Learning with Data & Label Privacy for thin clients

When MiniBatch SGD Meets SplitFed Learning:Convergence Analysis and Performance Evaluation

FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training

Splitfed learning without client-side synchronization: Analyzing client-side split network portion size to overall performance

AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networks

Accelerating Federated Learning Via Parallel Servers: A Theoretically Guaranteed Approach.

Federated learning with workload-aware client scheduling in heterogeneous systems