Abstract:One important approach of multiagent reinforcement learning (MARL) is equilibrium-based MARL, which is a combination of reinforcement learning and game theory. Most existing algorithms involve computationally expensive calculation of mixed strategy equilibria and require agents to replicate the other agents' value functions for equilibrium computing in each state. This is unrealistic since agents may not be willing to share such information due to privacy or safety concerns. This paper aims to develop novel and efficient MARL algorithms without the need for agents to share value functions. First, we adopt pure strategy equilibrium solution concepts instead of mixed strategy equilibria given that a mixed strategy equilibrium is often computationally expensive. In this paper, three types of pure strategy profiles are utilized as equilibrium solution concepts: pure strategy Nash equilibrium, equilibrium-dominating strategy profile, and nonstrict equilibrium-dominating strategy profile. The latter two solution concepts are strategy profiles from which agents can gain higher payoffs than one or more pure strategy Nash equilibria. Theoretical analysis shows that these strategy profiles are symmetric meta equilibria. Second, we propose a multistep negotiation process for finding pure strategy equilibria since value functions are not shared among agents. By putting these together, we propose a novel MARL algorithm called negotiation-based Q-learning (NegoQ). Experiments are first conducted in grid-world games, which are widely used to evaluate MARL algorithms. In these games, NegoQ learns equilibrium policies and runs significantly faster than existing MARL algorithms (correlated Q-learning and Nash Q-learning). Surprisingly, we find that NegoQ also performs well in team Markov games such as pursuit games, as compared with team-task-oriented MARL algorithms (such as friend Q-learning and distributed Q-learning).

Multi-Agent Trust Evaluation Model Based on Reinforcement Learning

Multi-Agent Trust Region Learning

Trust-based Consensus in Multi-Agent Reinforcement Learning Systems

Multiagent Reinforcement Learning: Methods, Trustworthiness, Applications in Intelligent Vehicles, and Challenges

A Game-Theoretic Approach to Multi-agent Trust Region Optimization.

Learning Mechanism of Trust Evaluation of MAS Based on Experience and Reputation

Fuzzy Set Theory-Based Trust Models in Multi-Agent Environment

Multiagent Reinforcement Learning with Unshared Value Functions.

A Survey of Trust and Reputation Systems in Multi-Agent Systems

Trust model of multi-agent system

Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges

Trust Evaluation Based on Community Discovery in Multi-Agent System

A Robust Model for Trust Evaluation during Interactions between Agents in a Sociable Environment

Revisiting Trust and Reputation in Multi-agent Systems

Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning

Trust and Reputation Mechanism in Multi-Agent System

SRL-Based Trust Predicting Model Used in Multi-Agent Systems

An Adaptive Dempster-Shafer Theory of Evidence Based Trust Model in Multiagent Systems

A Meta-Game Evaluation Framework for Deep Multiagent Reinforcement Learning

Three perspectives on multi-agent reinforcement learning

Reputation Evaluation Method for Open Multi-Agent Systems