Abstract:Inferring other agents’ mental states, such as their knowledge, beliefs and intentions, is thought to be essential for effective interactions with other agents. Recently, multi-agent systems trained via deep reinforcement learning have been shown to succeed in solving various tasks. Still, how each agent models or represents other agents in their environment remains unclear. In this work, we test whether deep reinforcement learning agents trained with the multi-agent deep deterministic policy gradient (MADDPG) algorithm explicitly represent other agents’ intentions (their specific aims or plans) during a task in which the agents have to coordinate the covering of different spots in a 2D environment. In particular, we tracked over time the performance of a linear decoder trained to predict the final targets of all agents from the hidden-layer activations of each agent’s neural network controller. We observed that the hidden layers of agents represented explicit information about other agents’ intentions, i.e., the target landmark the other agent ended up covering. We also performed a series of experiments in which some agents were replaced by others with fixed targets to test the levels of generalization of the trained agents. We noticed that during the training phase, the agents developed a preference for each landmark, which hindered generalization. To alleviate the above problem, we evaluated simple changes to the MADDPG training algorithm which lead to better generalization against unseen agents. Our method for confirming intention modeling in deep learning agents is simple to implement and can be used to improve the generalization of multi-agent systems in fields such as robotics, autonomous vehicles and smart cities.

Interpretability for Conditional Coordinated Behavior in Multi-Agent Reinforcement Learning

S2RL: DoWe Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

Distributed Multi-Agent Deep Reinforcement Learning for Robust Coordination against Noise

Towards Interpretable Reinforcement Learning Using Attention Augmented Agents

Joint Attention for Multi-Agent Coordination and Social Learning

AHAC: Actor Hierarchical Attention Critic for Multi-Agent Reinforcement Learning.

Attention-Guided Contrastive Role Representations for Multi-Agent Reinforcement Learning

Multi-agent Reinforcement Learning by the Actor-Critic Model with an Attention Interface

Cascaded Attention: Adaptive and Gated Graph Attention Network for Multiagent Reinforcement Learning

Attentive Relational State Representation in Decentralized Multiagent Reinforcement Learning.

Learning Multi-Agent Communication with Double Attentional Deep Reinforcement Learning

N$\text{A}^\text{2}$Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learning

Active Legibility in Multiagent Reinforcement Learning

Complementary Attention for Multi-Agent Reinforcement Learning.

Learning Attentional Communication with a Common Network for Multiagent Reinforcement Learning.

Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

Cooperative Action Decision Based on State Perception Similarity for Deep Reinforcement Learning

An Actor-Critic-Attention Mechanism for Deep Reinforcement Learning in Multi-view Environments

Do Deep Reinforcement Learning Agents Model Intentions?

DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning