Abstract:This study proposes a mechanism to handle stragglers in data‐parallel ML training by offloading the control of the SSP synchronization to the programmable data plane using a memory‐efficient data structure. This work also introduces an API for dynamic adjustment of synchronization barriers. Tested by injecting stragglers in a realistic ML workload, on both an emulated environment and real hardware, the system improves job completion time and waiting time with limited impact on model accuracy. Achieving high‐performance aggregation is essential to scaling data‐parallel distributed machine learning (ML) training. Recent research in in‐network computing has shown that offloading the aggregation to the network data plane can accelerate the aggregation process compared to traditional server‐only approaches, reducing the propagation delay and consequently speeding up distributed training. However, the existing literature on in‐network aggregation does not provide ways to deal with slower workers (called stragglers). The presence of stragglers can negatively impact distributed training, increasing the time it takes to complete. In this paper, we present Serene, an in‐network aggregation system capable of circumventing the effects of stragglers. Serene coordinates the ML workers to cooperate with a programmable switch using a hybrid synchronization approach where approaches can be changed dynamically. The synchronization can change dynamically through a control plane API that translates high‐level code into switch rules. Serene switch employs an efficient data structure for managing synchronization and a hot‐swapping mechanism to consistently change from one synchronization strategy to another. We implemented and evaluated a prototype using BMv2 and a Proof‐of‐Concept in a Tofino ASIC. We ran experiments with realistic ML workloads, including a neural network trained for image classification. Our results show that Serene can speed up training by up to 40% in emulation scenarios by reducing drastically the cumulative waiting time compared to a synchronous baseline.

Accelerating Distributed Learning in Non-Dedicated Environments

Semi-Dynamic Load Balancing: Efficient Distributed Learning in Non-Dedicated Environments

WBSP: Addressing Stragglers in Distributed Machine Learning with Worker-Busy Synchronous Parallel

Optimize Scheduling of Federated Learning on Battery-powered Mobile Devices

LBB: load-balanced batching for efficient distributed learning on heterogeneous GPU cluster

Locality-aware and Fault-tolerant Batching for Machine Learning on Distributed Datasets

Efficient Distributed Machine Learning with Trigger Driven Parallel Training.

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

No Worker Left (Too Far) Behind: Dynamic Hybrid Synchronization for In‐Network ML Aggregation

FSP: Towards Flexible Synchronous Parallel Frameworks for Distributed Machine Learning

Online Job Scheduling in Distributed Machine Learning Clusters

Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching

When Less is More: Achieving Faster Convergence in Distributed Edge Machine Learning

Straggler-aware Distributed Learning: Communication Computation Latency Trade-off

Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources

A Dynamic Weighting Strategy to Mitigate Worker Node Failure in Distributed Deep Learning

Online Scheduling Algorithm for Heterogeneous Distributed Machine Learning Jobs

Collaborative Learning Based Straggler Prevention in Large-Scale Distributed Computing Framework

A Hybrid Federated Learning Framework with Dynamic Task Allocation for Multi-Party Distributed Load Prediction

Cloudless-Training: A Framework to Improve Efficiency of Geo-Distributed ML Training

Latency Optimization for Blockchain-Empowered Federated Learning in Multi-Server Edge Computing