Can LLM-Augmented autonomous agents cooperate?, An evaluation of their cooperative capabilities through Melting Pot

Manuel Mosquera,Juan Sebastian Pinzon,Manuel Rios,Yesid Fonseca,Luis Felipe Giraldo,Nicanor Quijano,Ruben Manrique

2024-06-20

Abstract:As the field of AI continues to evolve, a significant dimension of this progression is the development of Large Language Models and their potential to enhance multi-agent artificial intelligence systems. This paper explores the cooperative capabilities of Large Language Model-augmented Autonomous Agents (LAAs) using the well-known Meltin Pot environments along with reference models such as GPT4 and GPT3.5. Preliminary results suggest that while these agents demonstrate a propensity for cooperation, they still struggle with effective collaboration in given environments, emphasizing the need for more robust architectures. The study's contributions include an abstraction layer to adapt Melting Pot game scenarios for LLMs, the implementation of a reusable architecture for LLM-mediated agent development - which includes short and long-term memories and different cognitive modules, and the evaluation of cooperation capabilities using a set of metrics tied to the Melting Pot's "Commons Harvest" game. The paper closes, by discussing the limitations of the current architectural framework and the potential of a new set of modules that fosters better cooperation among LAAs.

Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The paper aims to explore the cooperative capabilities of Large Language Model (LLM)-enhanced Autonomous Agents (LAAs). Specifically, the researchers utilize the Melting Pot environment and its reference models (such as GPT-4 and GPT-3.5) to evaluate these agents' collaborative abilities in multi-agent systems. Although preliminary results show that these agents exhibit a tendency to cooperate, they still struggle to achieve effective collaboration in specific environments, indicating the need for further development of more robust architectures.

The main contributions of the paper include:
1. Designing an abstraction layer to adapt to the Melting Pot game scenarios, allowing LLMs to better handle these scenarios.
2. Implementing a reusable architecture for developing autonomous agents based on LLMs, which includes short-term and long-term memory as well as different cognitive modules.
3. Using a set of metrics to evaluate the agents' cooperative abilities in the "Commons Harvest" game of Melting Pot.
4. Discussing the limitations of the current architectural framework and proposing potential improvements to the architecture to facilitate better cooperation among LAAs.