Abstract:Reinforcement learning (RL) has attracted much attention recently, as new and emerging AI-based applications are demanding the capabilities to intelligently react to environment changes. Unlike distributed deep neural network (DNN) training, the distributed RL training has its unique workload characteristics - it generates orders of magnitude more iterations with much smaller sized but more frequent gradient aggregations. More specifically, our study with typical RL algorithms shows that their distributed training is latency critical and that the network communication for gradient aggregation occupies up to 83.2% of the execution time of each training iteration. In this paper, we present iSwitch, an in-switch acceleration solution that moves the gradient aggregation from server nodes into the network switches, thus we can reduce the number of network hops for gradient aggregation. This not only reduces the end-to-end network latency for synchronous training, but also improves the convergence with faster weight updates for asynchronous training. Upon the in-switch accelerator, we further reduce the synchronization overhead by conducting on-the-fly gradient aggregation at the granularity of network packets rather than gradient vectors. Moreover, we rethink the distributed RL training algorithms and also propose a hierarchical aggregation mechanism to further increase the parallelism and scalability of the distributed RL training at rack scale. We implement iSwitch using a real-world programmable switch NetFPGA board. We extend the control and data plane of the programmable switch to support iSwitch without affecting its regular network functions. Compared with state-of-the-art distributed training approaches, iSwitch offers a system-level speedup of up to 3.66 for synchronous distributed training and 3.71 for asynchronous distributed training, while achieving better scalability.

Scalable On-Switch Rate Limiters for the Cloud.

Sra: Switch Resource Aggregation For Application Offloading In Programmable Networks

RateSheriff: Multipath Flow-aware and Resource Efficient Rate Limiter Placement for Data Center Networks.

RIFFLE: A Distributed Network Operating System with Re-configurability for Multiple Domains

Consistent and Fine-Grained Rule Update with In-Network Control for Distributed Rate Limiting

CMDRL: A Markovian Distributed Rate Limiting Algorithm in Cloud Networks

CLIP: Accelerating Features Deployment for Programmable Switch.

Vswitchlb: Stratified Load Balancing for Vswitch Efficiency in Data Centers

Poster: Triton: Accelerating Vswitch with Flexibility Through Hardware Assisting Not Bypassing Software

Low Latency Software Rate Limiters for Cloud Networks.

Cutting Long-Tail Latency of Routing Response in Software Defined Networks.

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds.

Accelerating distributed reinforcement learning with in-switch computing

Efficient Remote Radio Head Switching Scheme In Cloud Radio Access Network: A Load Balancing Perspective

Switch-Assistant Loss Recovery for RDMA Transport Control

SMART: Dual-channel Southbound Message Delivery in Clouds with Rate Estimation

A Resource Pooling Switch Architecture with High Performance Scheduler

Software-Defined Label Switching: Scalable Per-flow Control in SDN

SFP: Service Function Chain Provision on Programmable Switches for Cloud Tenants

Accessing Cloud with Disaggregated Software-Defined Router.

Programmable Switch as a Parallel Computing Device