Abstract: Learning to coordinate is a daunting problem in multi-agent reinforcement learning (MARL). Previous works have explored it from many facets, including cognition between agents, credit assignment, communication, expert demonstration, etc. However, less attention were paid to agents' decision structure and the hierarchy of coordination. In this paper, we explore the spatiotemporal structure of agents' decisions and consider the hierarchy of coordination from the perspective of multilevel emergence dynamics, based on which a novel approach, Learning to Advise and Learning from Advice (LALA), is proposed to improve MARL. Specifically, by distinguishing the hierarchy of coordination, we propose to enhance decision coordination at meso level with an advisor and leverage a policy discriminator to advise agents' learning at micro level. The advisor learns to aggregate decision information in both spatial and temporal domains and generates coordinated decisions by employing a spatiotemporal dual graph convolutional neural network with a task-oriented objective function. Each agent learns from the advice via a policy generative adversarial learning method where a discriminator distinguishes between the policies of the agent and the advisor and boosts both of them based on its judgement. Experimental results indicate the advantage of LALA over baseline approaches in terms of both learning efficiency and coordination capability. Coordination mechanism is investigated from the perspective of multilevel emergence dynamics and mutual information point of view, which provides a novel perspective and method to analyze and improve MARL algorithms.

Signal Instructed Coordination in Cooperative Multi-agent Reinforcement Learning

Learning Intra-group Cooperation in Multi-agent Systems.

S2rl

Coordination as inference in multi-agent reinforcement learning

S2RL: DoWe Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning

A Cooperative Multi-Agent Reinforcement Learning Method Based on Coordination Degree

Multi-Agent Incentive Communication via Decentralized Teammate Modeling

Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?

Learning multiagent coordination in the absence of communication channels

Multiagent Q-learning with Sub-Team Coordination.

Distributed Signal Control of Arterial Corridors Using Multi-Agent Deep Reinforcement Learning

Enhancing Multi-Agent Coordination through Common Operating Picture Integration

Multi-Agent Concentrative Coordination with Decentralized Task Representation

Coordinating Multi-Agent Reinforcement Learning Via Dual Collaborative Constraints

Learning to Advise and Learning from Advice in Cooperative Multiagent Reinforcement Learning.

Multi-Agent Curricula and Emergent Implicit Signaling

Enhancing cooperation by cognition differences and consistent representation in multi-agent reinforcement learning

Robust Multi-Agent Reinforcement Learning Driven by Correlated Equilibrium

Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey