Abstract:A computing cluster that interconnects multiple compute nodes is used to accelerate distributed reinforcement learning that uses DQN (Deep Q-Network). In distributed reinforcement learning, actor nodes acquire experiences by interacting with a given environment and a learner node optimizes the DQN model. When distributed reinforcement learning is used in practical applications such as robotics, we can assume that actor nodes are located in edge side while the learner node is located in cloud side. In this case, the long-haul communication between them imposes significant communication overheads. However, most prior works simply assume that actors and learner are located closely, and do not take the overheads into account. In this paper, we focus on the practical environment where the actors and learner are located remotely, and they interact via a buffer node that collects information from multiple actor nodes. We implement a prototype system in which the buffer and learner nodes are connected via a 25GbE (Gigabit Ethernet) switch and a 10km optical fiber cable. Although a replay memory functionality is closely associated with the learner side, in this paper we propose to combine the replay memory into the buffer node. In our experiments using the prototype system, the proposed approach is compared with an existing approach in terms of the training efficiency (i.e., training loss) and the transfer efficiency over the long-haul communication (i.e., average priority of transferred experiences). As a result, the training loss of the proposed approach is reduced to 26% of the existing approach, and the average priority is 3.92 times higher than the existing approach after the training loss is converged. These results demonstrate that the proposed approach can improve the training/communication efficiency compared with the existing approach in a practical system that imposes long-haul communication between the actors and learner.

Experimental Assessments of a AWGR Based Optical Switching Network for Distributed Reinforcement Learning Training

Experimental Demonstration of an AWGR-based Nanoseconds Optical Switching DCN.

Accelerating distributed reinforcement learning with in-switch computing

Performance Analysis of Distributed Optical Routing Network Based on Tunable Lasers and Cyclic AWG Router

Automatically Reconfigurable Optical Network for HPC System Based on Deep Reinforcement Learning

ABOI: AWGR-Based Optical Interconnects for Single-Wavelength and Multi-Wavelength

ODRAD: An optical wireless DCN dynamic-bandwidth reconfiguration with AWGR and deep reinforcement learning

Self-Routing Nonblocking WDM Switches Based on Arrayed Waveguide Grating.

RGAIA: a Reconfigurable AWGR Based Optical Data Center Network.

An Efficient Distributed Reinforcement Learning Architecture for Long-Haul Communication Between Actors and Learner

Enhanced-Performance Tunable Sources for Fast AWGR-based Optical Switching Data Center Networks

Generalized recursive network: a new architecture for self-routing non-blocking optical switch networks

Accelerate Distributed Deep Learning with a Fast Reconfigurable Optical Network

Deep learning-optical network routing algorithm based on wavelength continuity supervision

Performance Analysis of Various Multi-Granularity Switching Architectures and an On-Line RWA Algorithm

Deep Reinforcement Learning for Router Selection in Network with Heavy Traffic

ReSAW: a reconfigurable and picosecond-synchronized optical data center network based on an AWGR and the WR protocol

A Study of Modular Awgs for Large-Scale Optical Switching Systems

Reconfiguring Multicast Sessions in Elastic Optical Networks Adaptively with Graph-Aware Deep Reinforcement Learning

Optical Circuit Switching Enabled Reconfigurable HPC Network for Traffic Pattern

Waveband Grooming Based on Layered Auxiliary Graph in Multi-Domain Optical Networks