Abstract:One important approach of multiagent reinforcement learning (MARL) is equilibrium-based MARL, which is a combination of reinforcement learning and game theory. Most existing algorithms involve computationally expensive calculation of mixed strategy equilibria and require agents to replicate the other agents' value functions for equilibrium computing in each state. This is unrealistic since agents may not be willing to share such information due to privacy or safety concerns. This paper aims to develop novel and efficient MARL algorithms without the need for agents to share value functions. First, we adopt pure strategy equilibrium solution concepts instead of mixed strategy equilibria given that a mixed strategy equilibrium is often computationally expensive. In this paper, three types of pure strategy profiles are utilized as equilibrium solution concepts: pure strategy Nash equilibrium, equilibrium-dominating strategy profile, and nonstrict equilibrium-dominating strategy profile. The latter two solution concepts are strategy profiles from which agents can gain higher payoffs than one or more pure strategy Nash equilibria. Theoretical analysis shows that these strategy profiles are symmetric meta equilibria. Second, we propose a multistep negotiation process for finding pure strategy equilibria since value functions are not shared among agents. By putting these together, we propose a novel MARL algorithm called negotiation-based Q-learning (NegoQ). Experiments are first conducted in grid-world games, which are widely used to evaluate MARL algorithms. In these games, NegoQ learns equilibrium policies and runs significantly faster than existing MARL algorithms (correlated Q-learning and Nash Q-learning). Surprisingly, we find that NegoQ also performs well in team Markov games such as pursuit games, as compared with team-task-oriented MARL algorithms (such as friend Q-learning and distributed Q-learning).

Deep Multi-Agent Reinforcement Learning with Relevance Graphs

MAGNet: Multi-agent Graph Network for Deep Multi-agent Reinforcement Learning

Multiagent Reinforcement Learning With Heterogeneous Graph Attention Network

Graph Neural Networks with Model-based Reinforcement Learning for Multi-agent Systems

Very Large Scale Multi-Agent Reinforcement Learning with Graph Attention Mean Field

Graph MADDPG with RNN for multiagent cooperative environment

Partially Observable Mean Field Multi-Agent Reinforcement Learning Based on Graph-Attention

Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph

Graph Neural Network-based Multi-agent Reinforcement Learning for Resilient Distributed Coordination of Multi-Robot Systems

GAT-MF: Graph Attention Mean Field for Very Large Scale Multi-Agent Reinforcement Learning

Graph Neural Network Meets Multi-Agent Reinforcement Learning: Fundamentals, Applications, and Future Directions

An Overview of Multi-Agent Reinforcement Learning from Game Theoretical Perspective

Graphcomm: a graph neural network based method for multi-agent reinforcement learning

Multi-agent Reinforcement Learning with Deep Networks for Diverse Q-Vectors

Deep multiagent reinforcement learning: challenges and directions

Multiagent Reinforcement Learning With Graphical Mutual Information Maximization

DeepMAG: Deep reinforcement learning with multi-agent graphs for flexible job shop scheduling

A Cooperation Graph Approach for Multiagent Sparse Reward Reinforcement Learning

A Survey and Critique of Multiagent Deep Reinforcement Learning

Scalable and Transferable Reinforcement Learning for Multi-Agent Mixed Cooperative–Competitive Environments Based on Hierarchical Graph Attention

Multiagent Reinforcement Learning with Unshared Value Functions.