Abstract:Reinforcement learning (RL) has attracted much attention recently, as new and emerging AI-based applications are demanding the capabilities to intelligently react to environment changes. Unlike distributed deep neural network (DNN) training, the distributed RL training has its unique workload characteristics - it generates orders of magnitude more iterations with much smaller sized but more frequent gradient aggregations. More specifically, our study with typical RL algorithms shows that their distributed training is latency critical and that the network communication for gradient aggregation occupies up to 83.2% of the execution time of each training iteration. In this paper, we present iSwitch, an in-switch acceleration solution that moves the gradient aggregation from server nodes into the network switches, thus we can reduce the number of network hops for gradient aggregation. This not only reduces the end-to-end network latency for synchronous training, but also improves the convergence with faster weight updates for asynchronous training. Upon the in-switch accelerator, we further reduce the synchronization overhead by conducting on-the-fly gradient aggregation at the granularity of network packets rather than gradient vectors. Moreover, we rethink the distributed RL training algorithms and also propose a hierarchical aggregation mechanism to further increase the parallelism and scalability of the distributed RL training at rack scale. We implement iSwitch using a real-world programmable switch NetFPGA board. We extend the control and data plane of the programmable switch to support iSwitch without affecting its regular network functions. Compared with state-of-the-art distributed training approaches, iSwitch offers a system-level speedup of up to 3.66 for synchronous distributed training and 3.71 for asynchronous distributed training, while achieving better scalability.

LCDN: Providing Network Determinism with Low-Cost Switches

Design and FPGA Verification of a Novel Reliable Real-Time Data Transfer System

Large-Scale Deterministic Networks: Architecture, Enabling Technologies, Case Study and Future Directions

Large Scale Deterministic Networking: A Simulation Evaluation

High Capacity and Resilient Large-Scale Deterministic IP Networks

PL2: Towards Predictable Low Latency in Rack-Scale Networks

Lightweight Hard Real-Time Communication For Commodity Cluster Systems

Low-latency Deterministic Forwarding for Multi-modal Connection Using Time-Sensitive Networking.

Industrial Knee-jerk: In-Network Simultaneous Planning and Control on a TSN Switch

Work-in-Progress: Towards Real-Time IDS Via RNN and Programmable Switches Co-Designed Approach

A Guaranteed-Throughput Switch For Network-On-Chip

LND: a reliable multi-tier storage device in NOW

Ultra-Low Latency Wireless Communications for Deterministic Networking: A Cross-Layer Approach

Future Data Center Networking: from Low Latency to Deterministic Latency

Design of a Highly Reliable Redundant Network Card Supporting the IEEE1588 Protocol

An Efficient and Accurate Link Latency Monitoring Method for Low-Latency Software-Defined Networks

Longitudinal $\bar Λ_0$ polarization in heavy ion collisions as a probe for QGP formation

Accelerating distributed reinforcement learning with in-switch computing

Low power NoC architecture based dynamic reconfigurable system

DumbNet: a smart data center network fabric with dumb switches.

Low Jitter Scheduling with Redundancy Control for Input-Queued Switches