SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning

Qian Long,Fangwei Zhong,Mingdong Wu,Yizhou Wang,Song-Chun Zhu

2024-05-03

Abstract:Multi-agent systems (MAS) need to adaptively cope with dynamic environments, changing agent populations, and diverse tasks. However, most of the multi-agent systems cannot easily handle them, due to the complexity of the state and task space. The social impact theory regards the complex influencing factors as forces acting on an agent, emanating from the environment, other agents, and the agent's intrinsic motivation, referring to the social force. Inspired by this concept, we propose a novel gradient-based state representation for multi-agent reinforcement learning. To non-trivially model the social forces, we further introduce a data-driven method, where we employ denoising score matching to learn the social gradient fields (SocialGFs) from offline samples, e.g., the attractive or repulsive outcomes of each force. During interactions, the agents take actions based on the multi-dimensional gradients to maximize their own rewards. In practice, we integrate SocialGFs into the widely used multi-agent reinforcement learning algorithms, e.g., MAPPO. The empirical results reveal that SocialGFs offer four advantages for multi-agent systems: 1) they can be learned without requiring online interaction, 2) they demonstrate transferability across diverse tasks, 3) they facilitate credit assignment in challenging reward settings, and 4) they are scalable with the increasing number of agents.

Artificial Intelligence,Multiagent Systems

What problem does this paper attempt to address?

The paper aims to address the adaptability issues of Multi-Agent Systems (MAS) in dynamic environments, varying tasks, and different numbers and roles of agents. Most current multi-agent systems struggle to handle these complex situations, primarily due to the complexity of the state space and task space. To tackle this challenge, the researchers propose a novel gradient-based state representation method—Social Gradient Fields (SocialGFs)—for multi-agent reinforcement learning.

Specifically, this method is inspired by social force theory, viewing complex influencing factors as forces acting on agents, and learns these social gradient fields from offline samples using denoising score matching techniques. During interactions, agents take actions based on multidimensional gradients to maximize their own rewards. Experimental results show that SocialGFs have the following advantages:

1. Learning can be conducted without the need for online interactions;
2. Demonstrates good transferability between different tasks;
3. Aids in credit assignment in challenging reward settings;
4. Maintains scalability as the number of agents increases.

In summary, the paper improves the adaptability and generality of multi-agent systems by introducing SocialGFs, enabling them to operate effectively in various environments.