Abstract:Recent In-Network Aggregation (INA) solutions offload the all-reduce operation onto network switches to accelerate and scale distributed training (DT). On end hosts, these solutions build custom network stacks to replace the transport layer. The INA-oriented network stack cannot take advantage of the state-of-the-art performant transport layer implementation, and also causes complexity in system development and operation. We design a transport-transparent INA primitive named NetReduce for modern multi-rack data centers. NetReduce runs beneath the transport layer. The switch performs aggregation operations but preserves data transmission connections. The host uses RoCE as its transport layer to deliver gradient messages and receive aggregation results. NetReduce achieves performance gains from both INA and RoCE: linear scalability, traffic reduction, and bandwidth freeing-up from INA — high throughput, low latency, and low CPU overhead from RoCE. For jobs spanning several multi-GPU machines, we also devise parallel all-reduce based on NetReduce to make use of intra-machine and inter-machine bandwidth efficiently. We prototype NetReduce on an FPGA board attached to an Ethernet switch. We compare NetReduce with existing programmable switch-based solutions and justify the FPGA-based design choice. We evaluate NetReduce’s performance by training typical Deep Neural Network models on single-GPU and multi-GPU testbeds. NetReduce inter-operates with the existing Ethernet transport layer, is training-framework friendly, accelerates network-intensive DT jobs effectively (e.g., 70% for AlexNet), reduces CPU overheads (e.g., only one core for transmission), and is cost-effective (e.g., only 2.40% more capital expense and 0.68% more power consumption making 12.3-57.9% more performance acceleration).

ATP: In-network Aggregation for Multi-tenant Learning.

Accelerating Distributed Training with Collaborative In-Network Aggregation

In-Network Aggregation with Transport Transparency for Distributed Training

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Maximizing Aggregation Throughput for Distributed Training with Constrained In-Network Computing.

InArt: In-Network Aggregation with Route Selection for Accelerating Distributed Training

XAgg: Accelerating Heterogeneous Distributed Training Through XDP-Based Gradient Aggregation

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Accelerating Distributed Training Through In-network Aggregation with Idle Resources in Data Centers

Identifying Performance Bottleneck in Shared In-Network Aggregation During Distributed Training

AggTree: A Routing Tree with In-Network Aggregation for Distributed Training

Preemptive Switch Memory Usage to Accelerate Training Jobs with Shared In-Network Aggregation

ATP: Achieving Throughput Peak for DNN Training Via Smart GPU Memory Management

NQ/ATP: Architectural Support for Massive Aggregate Queries in Data Center Networks

Adaptive Training and Aggregation for Federated Learning in Multi-tier Computing Networks

Enabling Switch Memory Management for Distributed Training with In-Network Aggregation.

GACER: Granularity-Aware ConcurrEncy Regulation for Multi-Tenant Deep Learning

PARING: Joint Task Placement and Routing for Distributed Training with In-Network Aggregation

GRID: Gradient Routing with In-Network Aggregation for Distributed Training

Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models

Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregation