Abstract:StarCraft II is a challenging benchmark for AI agents due to the necessity of both precise micro level operations and strategic macro awareness. Previous works, such as Alphastar and SCC, achieve impressive performance on tackling StarCraft II , however, still exhibit deficiencies in long term strategic planning and strategy interpretability. Emerging large language model (LLM) agents, such as Voyage and MetaGPT, presents the immense potential in solving intricate tasks. Motivated by this, we aim to validate the capabilities of LLMs on StarCraft II, a highly complex RTS <a class="link-external link-http" href="http://game.To" rel="external noopener nofollow">this http URL</a> conveniently take full advantage of LLMs` reasoning abilities, we first develop textual StratCraft II environment, called TextStarCraft II, which LLM agent can interact. Secondly, we propose a Chain of Summarization method, including single frame summarization for processing raw observations and multi frame summarization for analyzing game information, providing command recommendations, and generating strategic decisions. Our experiment consists of two parts: first, an evaluation by human experts, which includes assessing the LLMs`s mastery of StarCraft II knowledge and the performance of LLM agents in the game; second, the in game performance of LLM agents, encompassing aspects like win rate and the impact of Chain of Summarization.Experiment results demonstrate that: 1. LLMs possess the relevant knowledge and complex planning abilities needed to address StarCraft II scenarios; 2. Human experts consider the performance of LLM agents to be close to that of an average player who has played StarCraft II for eight years; 3. LLM agents are capable of defeating the built in AI at the Harder(Lv5) difficulty level. We have open sourced the code and released demo videos of LLM agent playing StarCraft II.

Mastering Board Games by External and Internal Planning with Language Models

Can Large Language Models Play Games? A Case Study of A Self-Play Approach

Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games

Exploring and Benchmarking the Planning Capabilities of Large Language Models

Learning Chess With Language Models and Transformers

GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents

Explore the Reasoning Capability of LLMs in the Chess Testbed

Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models

Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models

Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach

Mastering Atari, Go, chess and shogi by planning with a learned model

From Code to Play: Benchmarking Program Search for Games Using Large Language Models

Strategic Features for General Games

Plan of Thoughts: Heuristic-Guided Problem Solving with Large Language Models

GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations

Improving Planning with Large Language Models: A Modular Agentic Architecture

Language Model Non-myopic Generation for Reasoning and Planning

Large Language Models on the Chessboard: A Study on ChatGPT's Formal Language Comprehension and Complex Reasoning Skills

LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

Strategist: Learning Strategic Skills by LLMs via Bi-Level Tree Search

Mastering the game of Stratego with model-free multiagent reinforcement learning