Abstract:The automatic evaluation of LLM-based agent intelligence is critical in developing advanced LLM-based agents. Although considerable effort has been devoted to developing human-annotated evaluation datasets, such as AlpacaEval, existing techniques are costly, time-consuming, and lack adaptability. In this paper, inspired by the popular language game ``Who is Spy'', we propose to use the word guessing game to assess the intelligence performance of LLMs. Given a word, the LLM is asked to describe the word and determine its identity (spy or not) based on its and other players' descriptions. Ideally, an advanced agent should possess the ability to accurately describe a given word using an aggressive description while concurrently maximizing confusion in the conservative description, enhancing its participation in the game. To this end, we first develop DEEP to evaluate LLMs' expression and disguising abilities. DEEP requires LLM to describe a word in aggressive and conservative modes. We then introduce SpyGame, an interactive multi-agent framework designed to assess LLMs' intelligence through participation in a competitive language-based board game. Incorporating multi-agent interaction, SpyGame requires the target LLM to possess linguistic skills and strategic thinking, providing a more comprehensive evaluation of LLMs' human-like cognitive abilities and adaptability in complex communication situations. The proposed evaluation framework is very easy to implement. We collected words from multiple sources, domains, and languages and used the proposed evaluation framework to conduct experiments. Extensive experiments demonstrate that the proposed DEEP and SpyGame effectively evaluate the capabilities of various LLMs, capturing their ability to adapt to novel situations and engage in strategic communication.

Playing Mastermind with Wordle-like Feedback

Infinite Wordle and the Mastermind numbers

Using Wordle for Learning to Design and Compare Strategies

Mastermind with a Linear Number of Queries

Playing Codenames with Language Graphs and Word Embeddings

Semantic, Orthographic, and Morphological Biases in Humans' Wordle Gameplay

AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback

Puzzle game: Prediction and Classification of Wordle Solution Words

Permutation Wordle

Prediction Model For Wordle Game Results With High Robustness

Ann wins the nonrepetitive game over four letters and the erase-repetition game over six letters

Effective Wordle Heuristics

Learning Character Strings via Mastermind Queries, With a Case Study Involving mtDNA

Selecting Seed Words for Wordle using Character Statistics

Optimal schemes for combinatorial query problems with integer feedback

ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games

Mastering Strategy Card Game (Legends of Code and Magic) via End-to-End Policy and Optimistic Smooth Fictitious Play

Let's Play Mahjong!

Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games

Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation

Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models