Abstract:Multi-agent reinforcement learning has emerged as a powerful framework for enabling agents to learn complex, coordinated behaviors but faces persistent challenges regarding its generalization, scalability and sample efficiency. Recent advancements have sought to alleviate those issues by embedding intrinsic symmetries of the systems in the policy. Yet, most dynamical systems exhibit little to no symmetries to exploit. This paper presents a novel framework for embedding extrinsic symmetries in multi-agent system dynamics that enables the use of symmetry-enhanced methods to address systems with insufficient intrinsic symmetries, expanding the scope of equivariant learning to a wide variety of MARL problems. Central to our framework is the Group Equivariant Graphormer, a group-modular architecture specifically designed for distributed swarming tasks. Extensive experiments on a swarm of symmetry-breaking quadrotors validate the effectiveness of our approach, showcasing its potential for improved generalization and zero-shot scalability. Our method achieves significant reductions in collision rates and enhances task success rates across a diverse range of scenarios and varying swarm sizes.

What problem does this paper attempt to address?

This paper attempts to solve the problems of generalization, scalability, and sample efficiency encountered by multi - agent reinforcement learning (MARL) when facing the learning of complex and coordinated behaviors. Specifically, the author points out that although multi - agent systems have great potential, their high - dimensional and decentralized nature makes them face many challenges in practical applications. ### Core Problems of the Paper 1. **Generalization**: The performance of multi - agent systems in different environments or tasks is not stable enough. 2. **Scalability**: As the number of agents increases, the performance of the system declines. 3. **Sample Efficiency**: A large amount of training data is required to achieve good performance. To solve these problems, the paper proposes a new framework, which expands the application range of equivariant learning by embedding extrinsic symmetries into the dynamics of multi - agent systems. This method aims to solve the problems of systems that lack intrinsic symmetries themselves. ### Main Contributions 1. **Formalize the Symmetry Properties of Multi - Robot Dynamical Systems**: Define the conditions for the optimal policy to become an equivariant function, and show the limitations of current equivariant RL methods for explicit symmetric systems. 2. **Method of Embedding Extrinsic Symmetries**: Propose a method of embedding extrinsic symmetries into the system, even if the system itself does not have these symmetries, thus expanding the applicability of the equivariant learning framework. 3. **Introduce the Group Equivariant Graphormer Architecture**: Design a modular network structure suitable for distributed swarm tasks and capable of handling different types of symmetries. ### Experimental Verification The paper verifies the effectiveness of this method through a large number of experiments, especially in the SE(3) symmetry - breaking quadrotor UAV swarm tasks, showing improved generalization ability and zero - sample scalability. The experimental results show that the policy embedded with SE(3) symmetry exhibits higher reward values, fewer collision times, and higher success rates in multiple scenarios. ### Summary By introducing extrinsic symmetries, this paper significantly improves the generalization and sample efficiency of multi - agent reinforcement learning, providing new ideas and methods for solving the control problems of complex multi - agent systems.

Symmetries-enhanced Multi-Agent Reinforcement Learning