Abstract: Learning to coordinate is a daunting problem in multi-agent reinforcement learning (MARL). Previous works have explored it from many facets, including cognition between agents, credit assignment, communication, expert demonstration, etc. However, less attention were paid to agents' decision structure and the hierarchy of coordination. In this paper, we explore the spatiotemporal structure of agents' decisions and consider the hierarchy of coordination from the perspective of multilevel emergence dynamics, based on which a novel approach, Learning to Advise and Learning from Advice (LALA), is proposed to improve MARL. Specifically, by distinguishing the hierarchy of coordination, we propose to enhance decision coordination at meso level with an advisor and leverage a policy discriminator to advise agents' learning at micro level. The advisor learns to aggregate decision information in both spatial and temporal domains and generates coordinated decisions by employing a spatiotemporal dual graph convolutional neural network with a task-oriented objective function. Each agent learns from the advice via a policy generative adversarial learning method where a discriminator distinguishes between the policies of the agent and the advisor and boosts both of them based on its judgement. Experimental results indicate the advantage of LALA over baseline approaches in terms of both learning efficiency and coordination capability. Coordination mechanism is investigated from the perspective of multilevel emergence dynamics and mutual information point of view, which provides a novel perspective and method to analyze and improve MARL algorithms.

Coordination Between Individual Agents in Multi-Agent Reinforcement Learning.

A Cooperative Multi-Agent Reinforcement Learning Method Based on Coordination Degree

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

S2RL: DoWe Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

S2rl

Coordination as inference in multi-agent reinforcement learning

Multi-Agent Concentrative Coordination with Decentralized Task Representation

Adaptive Individual Q-Learning-A Multiagent Reinforcement Learning Method for Coordination Optimization

Signal Instructed Coordination in Cooperative Multi-agent Reinforcement Learning

Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning

Learning to Advise and Learning from Advice in Cooperative Multiagent Reinforcement Learning.

Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning

TraCo: Learning Virtual Traffic Coordinator for Cooperation with Multi-Agent Reinforcement Learning.

Coordinating Multi-Agent Reinforcement Learning Via Dual Collaborative Constraints

Robust Multi-Agent Reinforcement Learning with Social Empowerment for Coordination and Communication

Multi-agent Continual Coordination Via Progressive Task Contextualization

Multiagent Continual Coordination via Progressive Task Contextualization

Relation-Aware Learning for Multi-Task Multi-Agent Cooperative Games

Attention-Guided Contrastive Role Representations for Multi-Agent Reinforcement Learning

Robust Multi-Agent Reinforcement Learning Driven by Correlated Equilibrium

Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph