Cooperation and Fairness in Multi-Agent Reinforcement Learning

Jasmine Jerry Aloor,Siddharth Nayak,Sydney Dolan,Hamsa Balakrishnan

2024-10-19

Abstract:Multi-agent systems are trained to maximize shared cost objectives, which typically reflect system-level efficiency. However, in the resource-constrained environments of mobility and transportation systems, efficiency may be achieved at the expense of fairness -- certain agents may incur significantly greater costs or lower rewards compared to others. Tasks could be distributed inequitably, leading to some agents receiving an unfair advantage while others incur disproportionately high costs. It is important to consider the tradeoffs between efficiency and fairness. We consider the problem of fair multi-agent navigation for a group of decentralized agents using multi-agent reinforcement learning (MARL). We consider the reciprocal of the coefficient of variation of the distances traveled by different agents as a measure of fairness and investigate whether agents can learn to be fair without significantly sacrificing efficiency (i.e., increasing the total distance traveled). We find that by training agents using min-max fair distance goal assignments along with a reward term that incentivizes fairness as they move towards their goals, the agents (1) learn a fair assignment of goals and (2) achieve almost perfect goal coverage in navigation scenarios using only local observations. For goal coverage scenarios, we find that, on average, our model yields a 14% improvement in efficiency and a 5% improvement in fairness over a baseline trained using random assignments. Furthermore, an average of 21% improvement in fairness can be achieved compared to a model trained on optimally efficient assignments; this increase in fairness comes at the expense of only a 7% decrease in efficiency. Finally, we extend our method to environments in which agents must complete coverage tasks in prescribed formations and show that it is possible to do so without tailoring the models to specific formation shapes.

Multiagent Systems,Artificial Intelligence,Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to achieve a balance between efficiency and fairness in a resource - constrained environment in multi - agent reinforcement learning (MARL). Specifically, the paper focuses on how, in multi - agent navigation tasks, to design an appropriate reward function so that agents can achieve fairness in task allocation without significantly sacrificing efficiency while completing tasks. It is mentioned in the paper that in resource - constrained environments such as traffic and transportation systems, although pursuing system - level efficiency is a common goal, this pursuit may come at the cost of sacrificing the fairness of some agents, resulting in some agents bearing excessive costs or receiving lower rewards. Therefore, the researchers propose to introduce a fairness metric (such as the reciprocal of the standard deviation of travel distance) and combine it with the min - max fair - distance target - allocation strategy to train agents to learn fair behavior patterns, thereby improving the fairness of task allocation while maintaining high efficiency.

The main contributions of the paper are as follows:
1. A new reward function is introduced, which can promote agents to learn fair behavior during navigation without affecting the efficiency of task completion.
2. A decentralized learning method is proposed, enabling agents to adaptively select targets according to local observations, thereby dynamically adjusting their behavior during execution.
3. It is shown that the proposed method can be extended to any number of agents and can form formations of any shape without retraining the model.