Abstract:In this empirical paper, we investigate how learning agents can be arranged in more efficient communication topologies for improved learning. This is an important problem because a common technique to improve speed and robustness of learning in deep reinforcement learning and many other machine learning algorithms is to run multiple learning agents in parallel. The standard communication architecture typically involves all agents intermittently communicating with each other (fully connected topology) or with a centralized server (star topology). Unfortunately, optimizing the topology of communication over the space of all possible graphs is a hard problem, so we borrow results from the networked optimization and collective intelligence literatures which suggest that certain families of network topologies can lead to strong improvements over fully-connected networks. We start by introducing alternative network topologies to DRL benchmark tasks under the Evolution Strategies paradigm which we call Network Evolution Strategies. We explore the relative performance of the four main graph families and observe that one such family (Erdos-Renyi random graphs) empirically outperforms all other families, including the de facto fully-connected communication topologies. Additionally, the use of alternative network topologies has a multiplicative performance effect: we observe that when 1000 learning agents are arranged in a carefully designed communication topology, they can compete with 3000 agents arranged in the de facto fully-connected topology. Overall, our work suggests that distributed machine learning algorithms would learn more efficiently if the communication topology between learning agents was optimized.

Optimal Network Topology for Effective Collective Response

Consensus Seeking in Multi-Agent Systems with an Active Leader and Communication Delays.

Influence of network topology on cooperative problem-solving systems

Collective Behavior of Generic Linear Agents Interacting by Dynamic Topology with General Structures

Collective Behavior for Group of Generic Linear Agents Interacting under Arbitrary Network Topology

Robustness of Multi-Agent Formation Based on Natural Connectivity

Optimal Topology of Multi-Agent Systems with Two Leaders: a Zero-Sum Game Perspective.

Evolving Neural Networks Reveal Emergent Collective Behavior from Minimal Agent Interactions

How to Organize your Deep Reinforcement Learning Agents: The Importance of Communication Topology

Consensus Formation Among Mobile Agents in Networks of Heterogeneous Interaction Venues

Reorganizing Complex Network to Improve Large-Scale Multiagent Teamwork

Dynamic Consensus and Formation: Fixed and Switching Topologies

Group Consensus of Multi-Agent Networks with Hybrid Interactions.

Effects of Strategy Switching and Network Topology on Decision-Making in Multi-Agent Systems.

Learning to Cooperate: Networks of Formation Agents with Switching Topologies

Complex Network Effects in Large Multi-robot Collaborate Information System

Leader Selection in Networks under Switching Topologies with Antagonistic Interactions.

Leader-Formation Control With Dynamic Interaction Topologies

Collective decision-making under changing social environments among agents adapted to sparse connectivity

Collective Behaviors of Mobile Robots Beyond the Nearest Neighbor Rules with Switching Topology

Decentralized adaptive consensus of multi-agent in networks with switching topologies