Abstract:There is a prevalence of multiagent reinforcement learning (MARL) methods that engage in centralized training. These methods rely on all the agents sharing various types of information, such as their actions or gradients, with a centralized trainer or each other during the learning. Subsequently, the methods produce agent policies whose prescriptions and performance are contingent on other agents engaging in behavior assumed by the centralized training. But, in many contexts, such as mixed or adversarial settings, this assumption may not be feasible. In this article, we present a new line of methods that relaxes this assumption and engages in decentralized training resulting in the agent's individual policy. The interactive advantage actor-critic (IA2C) maintains and updates beliefs over other agents' candidate behaviors based on (noisy) observations, thus enabling learning at the agent's own level. We also address MARL's prohibitive curse of dimensionality due to the presence of many agents in the system. Under assumptions of action anonymity and population homogeneity, often exhibited in practice, large numbers of other agents can be modeled aggregately by the count vectors of their actions instead of individual agent models. More importantly, we may model the distribution of these vectors and its update using the Dirichlet-multinomial model, which offers an elegant way to scale IA2C to many-agent systems. We evaluate the performance of the fully decentralized IA2C along with other known baselines on a novel Organization domain, which we introduce, and on instances of two existing domains. Experimental comparisons with prominent and recent baselines show that IA2C is more sample efficient, more robust to noise, and can scale to learning in systems with up to a hundred agents.

Attentional Opponent Modelling for Multi-agent Cooperation

Local Information Based Attentional Opponent Modelling In Multi-agent Reinforcement Learning

P-CMOMMT Algorithm For The Cooperative Multi-Robot Observation of Multiple Moving Targets

Model-Based Opponent Modeling

OM-TCN: A Dynamic and Agile Opponent Modeling Approach for Competitive Games

Decision-making with Speculative Opponent Models

Learning to Model Opponent Learning

Behavior Reasoning for Opponent Agents in Multi-Agent Learning Systems

Modeling opponent learning in multiagent repeated games

A brain-inspired theory of mind spiking neural network improves multi-agent cooperation and competition

Multi-agent Actor-Critic with Time Dynamical Opponent Model

Opponent portrait for multiagent reinforcement learning in competitive environment

Attention-Guided Contrastive Role Representations for Multi-Agent Reinforcement Learning

Learning Attentional Communication for Multi-Agent Cooperation

ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind

Emergence of Theory of Mind Collaboration in Multiagent Systems

Multi-Agent Combat in Non-Stationary Environments

Inverse Attention Agent for Multi-Agent System

Coordinate-Aligned Multi-Camera Collaboration for Active Multi-Object Tracking

Modeling and reinforcement learning in partially observable many-agent systems

Opponent Modeling in Deep Reinforcement Learning