Abstract:We address two major challenges of implicit coordination in multi-agent deep reinforcement learning: non-stationarity and exponential growth of state-action space, by combining Deep-Q Networks for policy learning with Nash equilibrium for action selection. Q-values proxy as payoffs in Nash settings, and mutual best responses define joint action selection. Coordination is implicit because multiple/no Nash equilibria are resolved deterministically. We demonstrate that knowledge of game type leads to an assumption of mirrored best responses and faster convergence than Nash-Q. Specifically, the Friend-or-Foe algorithm demonstrates signs of convergence to a Set Controller which jointly chooses actions for two agents. This encouraging given the highly unstable nature of decentralized coordination over joint actions. Inspired by the dueling network architecture, which decouples the Q-function into state and advantage streams, as well as residual networks, we learn both a single and joint agent representation, and merge them via element-wise addition. This simplifies coordination by recasting it is as learning a residual function. We also draw high level comparative insights on key MADRL and game theoretic variables: competitive vs. cooperative, asynchronous vs. parallel learning, greedy versus socially optimal Nash equilibria tie breaking, and strategies for the no Nash equilibrium case. We evaluate on 3 custom environments written in Python using OpenAI Gym: a Predator Prey environment, an alternating Warehouse environment, and a Synchronization environment. Each environment requires successively more coordination to achieve positive rewards.

Zonation Method for Efficient Training of Collaborative Multi-Agent Reinforcement Learning in Double Snake Game

Learning to Cooperate: Application of Deep Reinforcement Learning for Online AGV Path Finding.

Learning Intra-group Cooperation in Multi-agent Systems.

Mapless Collaborative Navigation for a Multi-Robot System Based on the Deep Reinforcement Learning

Mastering the Game of 3v3 Snakes with Rule-Enhanced Multi-Agent Reinforcement Learning

Multi-agent Collaboration for Feasible Collaborative Behavior Construction and Evaluation

Decomposing Shared Networks for Separate Cooperation with Multi-Agent Reinforcement Learning

Weighted Double Deep Multiagent Reinforcement Learning in Stochastic Cooperative Environments

Deep reinforcement learning algorithm based on multi-agent parallelism and its application in game environment

Coordinating Multi-Agent Reinforcement Learning Via Dual Collaborative Constraints

Cooperative Learning of Multi-Agent Systems Via Reinforcement Learning

Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation

Resolving Implicit Coordination in Multi-Agent Deep Reinforcement Learning with Deep Q-Networks & Game Theory

Learning Multi-Agent Cooperation via Considering Actions of Teammates

A Memory Efficient Deep Reinforcement Learning Approach For Snake Game Autonomous Agents

Generating intelligent agent behaviors in multi-agent game AI using deep reinforcement learning algorithm

Collaboration of AI Agents via Cooperative Multi-Agent Deep Reinforcement Learning

Collaborative Deep Reinforcement Learning for Joint Object Search

HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism

Deep hierarchical reinforcement learning for collaborative object transportation by heterogeneous agents