Abstract:Can Large Language Models (LLMs) simulate human behavior in complex environments? LLMs have recently been shown to exhibit advanced reasoning skills but much of NLP evaluation still relies on static benchmarks. Answering this requires evaluation environments that probe strategic reasoning in competitive, dynamic scenarios that involve long-term planning. We introduce AucArena, a novel simulation environment for evaluating LLMs within auctions, a setting chosen for being highly unpredictable and involving many skills related to resource and risk management, while also being easy to evaluate. We conduct several controlled simulations using state-of-the-art LLMs as bidding agents. We find that through simple prompting, LLMs do indeed demonstrate many of the skills needed for effectively engaging in auctions (e.g., managing budget, adhering to long-term goals and priorities), skills that we find can be sharpened by explicitly encouraging models to be adaptive and observe strategies in past auctions. These results are significant as they show the potential of using LLM agents to model intricate social dynamics, especially in competitive settings. However, we also observe considerable variability in the capabilities of individual LLMs. Notably, even our most advanced models (GPT-4) are occasionally surpassed by heuristic baselines and human agents, highlighting the potential for further improvements in the design of LLM agents and the important role that our simulation environment can play in further testing and refining agent architectures.

Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction

Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Game

Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf

Enhance Reasoning for Large Language Models in the Game Werewolf

Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game

An Implementation of Werewolf Agent That does not Truly Trust LLMs

Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

GameEval: Evaluating LLMs on Conversational Games

Logic-Enhanced Language Model Agents for Trustworthy Social Simulations

Enhancing Dialogue Generation in Werewolf Game Through Situation Analysis and Persuasion Strategies

Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks

From Text to Tactic: Evaluating LLMs Playing the Game of Avalon

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

LLMs are Superior Feedback Providers: Bootstrapping Reasoning for Lie Detection with Self-Generated Feedback

How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments

Playing the Werewolf game with artificial intelligence for language understanding

Adversarial Multi-Agent Evaluation of Large Language Models through Iterative Debates

AvalonBench: Evaluating LLMs Playing the Game of Avalon

An Empirical Analysis on Large Language Models in Debate Evaluation