Abstract:Existing multi-agent coordination techniques are often fragile and vulnerable to anomalies such as agent attrition and communication disturbances, which are quite common in the real-world deployment of systems like field robotics. To better prepare these systems for the real world, we present a graph neural network (GNN)-based multi-agent reinforcement learning (MARL) method for resilient distributed coordination of a multi-robot system. Our method, Multi-Agent Graph Embedding-based Coordination (MAGEC), is trained using multi-agent proximal policy optimization (PPO) and enables distributed coordination around global objectives under agent attrition, partial observability, and limited or disturbed communications. We use a multi-robot patrolling scenario to demonstrate our MAGEC method in a ROS 2-based simulator and then compare its performance with prior coordination approaches. Results demonstrate that MAGEC outperforms existing methods in several experiments involving agent attrition and communication disturbance, and provides competitive results in scenarios without such anomalies.

What problem does this paper attempt to address?

This paper proposes an approach called Multi-Agent Graph Embedding-based Coordination (MAGEC) which is based on Graph Neural Networks (GNN) for Multi-Agent Reinforcement Learning (MARL). MAGEC aims to enhance the robust distributed coordination of multi-robot systems. Existing multi-agent coordination techniques are often fragile when facing exceptional situations such as agent loss and communication interference. MAGEC utilizes Proximal Policy Optimization (PPO) for multi-agent policy optimization during training, enabling distributed coordination under global objectives, even in the presence of agent loss, partial observability, and limited or interfered communication conditions. The core of MAGEC lies in utilizing GNN to process local information in the environment and learning graph embeddings through the message passing process to capture the attributes of nodes and edges, as well as their complex relationships. This approach aims to address the coordination challenges of multi-robot systems in the presence of agent loss and communication problems, especially in real-world environments with frequent interferences such as battlefields. In the paper, MAGEC is validated through simulations in a multi-robot patrol scenario and compared with existing coordination methods. The results demonstrate that MAGEC outperforms in various experiments involving loss and communication interference, and remains competitive in scenarios without such exceptional situations. Furthermore, the paper discusses related work including the application of GNN in multi-agent reinforcement learning, multi-agent patrol algorithms, and provides detailed descriptions of problem definition, design approach, training environment, and evaluation methods. Overall, MAGEC offers a novel and interference-resilient multi-agent coordination strategy, aiming to improve the practicality and adaptability of multi-robot systems.

Graph Neural Network-based Multi-agent Reinforcement Learning for Resilient Distributed Coordination of Multi-Robot Systems

A Graph-Based Soft Actor Critic Approach in Multi-Agent Reinforcement Learning

Graph Soft Actor-Critic Reinforcement Learning for Large-Scale Distributed Multirobot Coordination

Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning

Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph

Graph Neural Network Meets Multi-Agent Reinforcement Learning: Fundamentals, Applications, and Future Directions

Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigation

A Cooperation Graph Approach for Multiagent Sparse Reward Reinforcement Learning

Inferring Latent Temporal Sparse Coordination Graph for Multi-Agent Reinforcement Learning

Multiagent Reinforcement Learning With Heterogeneous Graph Attention Network

Graph Neural Networks with Model-based Reinforcement Learning for Multi-agent Systems

Scaling Up Multiagent Reinforcement Learning for Robotic Systems: Learn an Adaptive Sparse Communication Graph

Graphcomm: a graph neural network based method for multi-agent reinforcement learning

Proximal Policy Optimization Based Decentralized Networked Multi-Agent Reinforcement Learning

Graph-based Multi-Agent Reinforcement Learning for Collaborative Search and Tracking of Multiple UAVs

MAGNet: Multi-agent Graph Network for Deep Multi-agent Reinforcement Learning

GAMA: Graph Attention Multi-agent reinforcement learning algorithm for cooperation

Multi-Agent Deep Reinforcement Learning For Persistent Monitoring With Sensing, Communication, and Localization Constraints

Entropy Enhanced Multi-Agent Coordination Based on Hierarchical Graph Learning for Continuous Action Space

GCMA: an Adaptive Multi-Agent Reinforcement Learning Framework with Group Communication for Complex and Similar Tasks Coordination

Cooperative Policy Learning with Pre-trained Heterogeneous Observation Representations