Abstract:While large language models perform well on a range of complex tasks (e.g., text generation, question answering, summarization), robust multi-step planning and reasoning remains a considerable challenge for them. In this paper we show that search-based planning can significantly improve LLMs' playing strength across several board games (Chess, Fischer Random / Chess960, Connect Four, and Hex). We introduce, compare and contrast two major approaches: In external search, the model guides Monte Carlo Tree Search (MCTS) rollouts and evaluations without calls to an external engine, and in internal search, the model directly generates in-context a linearized tree of potential futures and a resulting final choice. Both build on a language model pre-trained on relevant domain knowledge, capturing the transition and value functions across these games. We find that our pre-training method minimizes hallucinations, as our model is highly accurate regarding state prediction and legal moves. Additionally, both internal and external search indeed improve win-rates against state-of-the-art bots, even reaching Grandmaster-level performance in chess while operating on a similar move count search budget per decision as human Grandmasters. The way we combine search with domain knowledge is not specific to board games, suggesting direct extensions into more general language model inference and training techniques.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: Large Language Models (LLMs) still face enormous challenges in their ability to perform robust multi - step planning and reasoning in complex tasks. Specifically, although LLMs perform well in tasks such as text generation, question - answering, and summarization, in tasks that require in - depth planning and reasoning, such as chess - like games (chess, Fischer Random/Chess960, Connect Four, Hex), their performance is not satisfactory and they are prone to hallucinations and reasoning errors. To solve this problem, the paper proposes two main methods: 1. **External Search**: Through Monte Carlo Tree Search (MCTS), the language model guides the move selection and evaluation during the simulation process without invoking an external engine. This enables the model to conduct effective search and planning without relying on external tools. 2. **Internal Search**: The language model directly generates a linearized tree of potential future states in the context and finally makes a choice. This method internalizes the search process into the model itself, reducing the dependence on external tools. The main contributions of the paper include: - **Contribution 1: MAV Model**: A Transformer model - the Multi - Action - Value (MAV) model - is pre - trained. This model can serve as a world model, a value function, and a policy function simultaneously, applicable to multiple perfect - information chess - like games. - **Contribution 2: External Search**: Using the MAV model as a value/implicit policy and transition function in an external MCTS controller significantly improves the game strength, reaching the master level. - **Contribution 3: Internal Search**: The search process is directly distilled into the language model, enabling the model to infer the search process, and its performance improves as the given search budget increases. Through these methods, the paper demonstrates how to combine search with domain knowledge to improve the reasoning ability of LLMs, and points out that this method is not limited to chess - like games but can also be extended to broader language model reasoning and training techniques.

Mastering Board Games by External and Internal Planning with Language Models