LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments

Junzhe Chen,Xuming Hu,Shuodi Liu,Shiyu Huang,Wei-Wei Tu,Zhaofeng He,Lijie Wen

2024-02-26

Abstract:Recent advancements in large language models (LLMs) have revealed their potential for achieving autonomous agents possessing human-level intelligence. However, existing benchmarks for evaluating LLM Agents either use static datasets, potentially leading to data leakage or focus only on single-agent scenarios, overlooking the complexities of multi-agent interactions. There is a lack of a benchmark that evaluates the diverse capabilities of LLM agents in multi-agent, dynamic environments. To this end, we introduce LLMArena, a novel and easily extensible framework for evaluating the diverse capabilities of LLM in multi-agent dynamic environments. LLMArena encompasses seven distinct gaming environments, employing Trueskill scoring to assess crucial abilities in LLM agents, including spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. We conduct an extensive experiment and human evaluation among different sizes and types of LLMs, showing that LLMs still have a significant journey ahead in their development towards becoming fully autonomous agents, especially in opponent modeling and team collaboration. We hope LLMArena could guide future research towards enhancing these capabilities in LLMs, ultimately leading to more sophisticated and practical applications in dynamic, multi-agent settings. The code and data will be available.

Computer Science

What problem does this paper attempt to address?

The paper primarily aims to address the issue of evaluating the capabilities of large language models (LLMs) in dynamic multi-agent environments. Existing evaluation benchmarks either use static datasets that may lead to data leakage or focus solely on single-agent scenarios, neglecting the complexity of multi-agent interactions. Therefore, the paper proposes a new evaluation framework—LLMA RENA, designed to assess the capabilities of large language models as agents in multi-agent systems.

LLMA RENA includes seven different game environments, through which the capabilities of large language models can be comprehensively evaluated in aspects such as spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, opponent modeling, and team collaboration. To more accurately assess the performance of these models, the study employs the TrueSkill™ rating system, which not only measures win rates but also considers skill level differences, providing a more comprehensive evaluation result.

The experimental section showcases the performance of 14 large language models of different scales and types across the seven environments and analyzes the relationship between model parameter scale and performance. The results indicate that as the scale of model parameters increases, the overall performance of the models improves, but challenges remain in tasks requiring numerical reasoning, opponent modeling, and team collaboration.

In summary, by proposing the LLMA RENA evaluation framework, the paper provides a direction for future research on the key capabilities of large language models in dynamic multi-agent environments, promoting the development of such models in real-world applications.