Abstract:We explore the hypothesis that LLMs, such as GPT-3.5 and GPT-4, possess broader cognitive functions, particularly in non-linguistic domains. Our approach extends beyond standard linguistic benchmarks by incorporating games like Tic-Tac-Toe, Connect Four, and Battleship, encoded via ASCII, to assess strategic thinking and decision-making. To evaluate the models' ability to generalize beyond their training data, we introduce two additional games. The first game, LEGO Connect Language (LCL), tests the models' capacity to understand spatial logic and follow assembly instructions. The second game, the game of shapes, challenges the models to identify shapes represented by 1s within a matrix of zeros, further testing their spatial reasoning skills. This "show, don't tell" strategy uses games instead of simply querying the models. Our results show that despite their proficiency on standard benchmarks, GPT-3.5 and GPT-4's abilities to play and reason about fully observable games without pre-training is mediocre. Both models fail to anticipate losing moves in Tic-Tac-Toe and Connect Four, and they are unable to play Battleship correctly. While GPT-4 shows some success in the game of shapes, both models fail at the assembly tasks presented in the LCL game. These results suggest that while GPT models can emulate conversational proficiency and basic rule comprehension, their performance in strategic gameplay and spatial reasoning tasks is very limited. Importantly, this reveals a blind spot in current LLM benchmarks that we highlight with our gameplay benchmark suite ChildPlay (<a class="link-external link-https" href="https://github.com/child-play-neurips/child-play" rel="external noopener nofollow">this https URL</a>). Our findings provide a cautionary tale about claims of emergent intelligence and reasoning capabilities of LLMs that are roughly the size of GPT-3.5 and GPT-4.

The Game of Tetris in Machine Learning

Playing Tetris with Reinforcement Learning

Apply Ant Colony Optimization to Tetris.

Deep reinforcement learning in playing Tetris with robotic arm experiment

Zone In Not Out! The Key to Winning High-Level Tetris

AGB stars in extragalactic systems

AlphaZero Gomoku

Implementation of Tetris as a Model Counter

Using Genetic Algorithm to Solve Puzzle Games: A Review

The application of artificial intelligence in board games

Reinforcement Learning: Playing Tic-Tac-Toe

AlphaZero-Inspired Game Learning: Faster Training by Using MCTS Only at Test Time

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

On Reinforcement Learning for the Game of 2048

Meta-Learning in Games

The Tsetlin Machine -- A Game Theoretic Bandit Driven Approach to Optimal Pattern Recognition with Propositional Logic

Learning in Games: a Systematic Review

Logic-based AI for Interpretable Board Game Winner Prediction with Tsetlin Machine

Reinforcement Learning For Constraint Satisfaction Game Agents (15-Puzzle, Minesweeper, 2048, and Sudoku)

Deep Apprenticeship Learning for Playing Games

Show, Don't Tell: Evaluating Large Language Models Beyond Textual Understanding with ChildPlay