Abstract:Multi-agent reinforcement learning (MARL) suffers from the non-stationarity problem, which is the ever-changing targets at every iteration when multiple agents update their policies at the same time. Starting from first principle, in this paper, we manage to solve the non-stationarity problem by proposing bidirectional action-dependent Q-learning (ACE). Central to the development of ACE is the sequential decision making process wherein only one agent is allowed to take action at one time. Within this process, each agent maximizes its value function given the actions taken by the preceding agents at the inference stage. In the learning phase, each agent minimizes the TD error that is dependent on how the subsequent agents have reacted to their chosen action. Given the design of bidirectional dependency, ACE effectively turns a multiagent MDP into a single-agent MDP. We implement the ACE framework by identifying the proper network representation to formulate the action dependency, so that the sequential decision process is computed implicitly in one forward pass. To validate ACE, we compare it with strong baselines on two MARL benchmarks. Empirical experiments demonstrate that ACE outperforms the state-of-the-art algorithms on Google Research Football and StarCraft Multi-Agent Challenge by a large margin. In particular, on SMAC tasks, ACE achieves 100% success rate on almost all the hard and super hard maps. We further study extensive research problems regarding ACE, including extension, generalization and practicability.

A Study of Reinforcement Learning in a New Multiagent Domain

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

Reinforcement learning based on multi-agent in robocup

An Improved Approach Towards Multi-Agent Pursuit–Evasion Game Decision-Making Using Deep Reinforcement Learning

Multiple rewards fuzzy reinforcement learning algorithm in RoboCup environment

Application of multi-agent reinforcement learning to RoboCup

Learning RoboCup-Keepaway with Kernels

REINFORCEMENT LEARNING-BASED HIGH-LEVEL BALL-STEALING STRATEGY FOR ROBOCUP KEEPAWAY

An optimal strategy learning for RoboCup in continuous state space

Prediction Learning In Multi Agent Systems

Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient

Cooperative Strategy Learning in Multi-Agent Environment with Continuous State Space

Multi-Robot Real-time Game Strategy Learning Based on Deep Reinforcement Learning.

Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces

MARLadona -- Towards Cooperative Team Play Using Multi-Agent Reinforcement Learning

A new multi-agent reinforcement learning approach

Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning

ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency

Fuzzy CMAC with automatic state partition for reinforcementlearning

Cooperative multi-agent game based on reinforcement learning

Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer