Abstract:This study proposes a mechanism to handle stragglers in data‐parallel ML training by offloading the control of the SSP synchronization to the programmable data plane using a memory‐efficient data structure. This work also introduces an API for dynamic adjustment of synchronization barriers. Tested by injecting stragglers in a realistic ML workload, on both an emulated environment and real hardware, the system improves job completion time and waiting time with limited impact on model accuracy. Achieving high‐performance aggregation is essential to scaling data‐parallel distributed machine learning (ML) training. Recent research in in‐network computing has shown that offloading the aggregation to the network data plane can accelerate the aggregation process compared to traditional server‐only approaches, reducing the propagation delay and consequently speeding up distributed training. However, the existing literature on in‐network aggregation does not provide ways to deal with slower workers (called stragglers). The presence of stragglers can negatively impact distributed training, increasing the time it takes to complete. In this paper, we present Serene, an in‐network aggregation system capable of circumventing the effects of stragglers. Serene coordinates the ML workers to cooperate with a programmable switch using a hybrid synchronization approach where approaches can be changed dynamically. The synchronization can change dynamically through a control plane API that translates high‐level code into switch rules. Serene switch employs an efficient data structure for managing synchronization and a hot‐swapping mechanism to consistently change from one synchronization strategy to another. We implemented and evaluated a prototype using BMv2 and a Proof‐of‐Concept in a Tofino ASIC. We ran experiments with realistic ML workloads, including a neural network trained for image classification. Our results show that Serene can speed up training by up to 40% in emulation scenarios by reducing drastically the cumulative waiting time compared to a synchronous baseline.

Preemptive Switch Memory Usage to Accelerate Training Jobs with Shared In-Network Aggregation

Efficient Data-Plane Memory Scheduling for In-Network Aggregation

Enabling Switch Memory Management for Distributed Training with In-Network Aggregation.

Accelerating Distributed Training with Collaborative In-Network Aggregation

Sra: Switch Resource Aggregation For Application Offloading In Programmable Networks

Training Job Placement in Clusters with Statistical In-Network Aggregation

In-Network Aggregation with Transport Transparency for Distributed Training

AggTree: A Routing Tree with In-Network Aggregation for Distributed Training

PARING: Joint Task Placement and Routing for Distributed Training with In-Network Aggregation

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Maximizing Aggregation Throughput for Distributed Training with Constrained In-Network Computing.

InArt: In-Network Aggregation with Route Selection for Accelerating Distributed Training

ATP: In-network Aggregation for Multi-tenant Learning.

Identifying Performance Bottleneck in Shared In-Network Aggregation During Distributed Training

XAgg: Accelerating Heterogeneous Distributed Training Through XDP-Based Gradient Aggregation

Ada-Grouper: Accelerating Pipeline Parallelism in Preempted Network by Adaptive Group-Scheduling for Micro-Batches

Accelerating distributed reinforcement learning with in-switch computing

Accelerating Distributed Training Through In-network Aggregation with Idle Resources in Data Centers

D2T: Dynamic Dual Threshold Policy of Shared-Memory in Data Center Switches

A Deep Learning Dataloader with Shared Data Preparation

No Worker Left (Too Far) Behind: Dynamic Hybrid Synchronization for In‐Network ML Aggregation