Abstract:We explore the hypothesis that LLMs, such as GPT-3.5 and GPT-4, possess broader cognitive functions, particularly in non-linguistic domains. Our approach extends beyond standard linguistic benchmarks by incorporating games like Tic-Tac-Toe, Connect Four, and Battleship, encoded via ASCII, to assess strategic thinking and decision-making. To evaluate the models' ability to generalize beyond their training data, we introduce two additional games. The first game, LEGO Connect Language (LCL), tests the models' capacity to understand spatial logic and follow assembly instructions. The second game, the game of shapes, challenges the models to identify shapes represented by 1s within a matrix of zeros, further testing their spatial reasoning skills. This "show, don't tell" strategy uses games instead of simply querying the models. Our results show that despite their proficiency on standard benchmarks, GPT-3.5 and GPT-4's abilities to play and reason about fully observable games without pre-training is mediocre. Both models fail to anticipate losing moves in Tic-Tac-Toe and Connect Four, and they are unable to play Battleship correctly. While GPT-4 shows some success in the game of shapes, both models fail at the assembly tasks presented in the LCL game. These results suggest that while GPT models can emulate conversational proficiency and basic rule comprehension, their performance in strategic gameplay and spatial reasoning tasks is very limited. Importantly, this reveals a blind spot in current LLM benchmarks that we highlight with our gameplay benchmark suite ChildPlay (<a class="link-external link-https" href="https://github.com/child-play-neurips/child-play" rel="external noopener nofollow">this https URL</a>). Our findings provide a cautionary tale about claims of emergent intelligence and reasoning capabilities of LLMs that are roughly the size of GPT-3.5 and GPT-4.

Playing Games With GPT: What Can We Learn About a Large Language Model From Canonical Strategic Games?

Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?

Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Games

Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis

Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing

GPT-3.5 altruistic advice is sensitive to reciprocal concerns but not to strategic risk

Show, Don't Tell: Evaluating Large Language Models Beyond Textual Understanding with ChildPlay

Large Language Models Playing Mixed Strategy Nash Equilibrium Games

The Machine Psychology of Cooperation: Can GPT models operationalise prompts for altruism, cooperation, competitiveness and selfishness in economic games?

Strategic behavior of large language models and the role of game structure versus contextual framing

Assessing Large Language Models' ability to predict how humans balance self-interest and the interest of others

The Good, the Bad, and the Hulk-like GPT: Analyzing Emotional Decisions of Large Language Models in Cooperation and Bargaining Games

Benchmarking Large Language Model (LLM) Performance for Game Playing via Tic-Tac-Toe

GPT in Game Theory Experiments

Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions

Diminished diversity-of-thought in a standard large language model

Effect of Private Deliberation: Deception of Large Language Models in Game Play

Using cognitive psychology to understand GPT-3

Can Machines Think Like Humans? A Behavioral Evaluation of LLM-Agents in Dictator Games

GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations

How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments