The Mathematical Game

Marc Pierre,Quentin Cohen-Solal,Tristan Cazenave

2023-09-22

Abstract:Monte Carlo Tree Search can be used for automated theorem proving. Holophrasm is a neural theorem prover using MCTS combined with neural networks for the policy and the evaluation. In this paper we propose to improve the performance of the Holophrasm theorem prover using other game tree search algorithms.

Artificial Intelligence

What problem does this paper attempt to address?

Proof Number Based Monte-Carlo Tree Search

Jakub Kowalski,Elliot Doe,Mark H. M. Winands,Daniel Górski,Dennis J. N. J. Soemers

2024-05-29

Abstract:This paper proposes a new game-search algorithm, PN-MCTS, which combines Monte-Carlo Tree Search (MCTS) and Proof-Number Search (PNS). These two algorithms have been successfully applied for decision making in a range of domains. We define three areas where the additional knowledge provided by the proof and disproof numbers gathered in MCTS trees might be used: final move selection, solving subtrees, and the UCB1 selection mechanism. We test all possible combinations on different time settings, playing against vanilla UCT on several games: Lines of Action ($7$$\times$$7$ and $8$$\times$$8$ board sizes), MiniShogi, Knightthrough, and Awari. Furthermore, we extend this new algorithm to properly address games with draws, like Awari, by adding an additional layer of PNS on top of the MCTS tree. The experiments show that PN-MCTS is able to outperform MCTS in all tested game domains, achieving win rates up to 96.2% for Lines of Action.

Artificial Intelligence
Belief-state Monte-Carlo Tree Search for Phantom Games

Jiao Wang,Tan Zhu,Hongye Li,Chu-Hsuan Hsueh,I-Chen Wu

DOI: https://doi.org/10.1109/tciaig.2017.2734067

2015-01-01

Abstract:Playing games with imperfect information is a very challenging issue in AI field due to its high complexity. Phantom game is a kind of such games, which usually has a large game-tree complexity and has little research achievements until now. In Phantom games, rational human players commonly select actions according to their beliefs in the game, which can be represented as a concept of belie f-state. To the best of our knowledge, our paper is the first article to incorporate belief-states in the Monte-Carlo Tree Search, and the proposed algorithm is named BS-MCTS (Belief-state Monte-Carlo Tree Search). In BS-MCTS, a belief-state tree, in which each node is a belief-state, is constructed and the search procedure is in accordance with beliefs updated by heuristic search information. We also present two novel implementations in the belief learning, that are Opponent Guessing and Opponent Predicting, concerning the probability on the possible states and on future actions of the opponent respectively. To prove the effectiveness of our algorithm, BS-MCTS is applied to Phantom Tic-Tac-Toe and Phantom Go against other Monte-Carlo methods. The experimental results demonstrate that our method is outstanding and advanced. Moreover, based on BS-MCTS, our Phantom Go program had consecutively won three championships in Chinese National Tournaments.
Monte Carlo Game Solver

Tristan Cazenave

DOI: https://doi.org/10.48550/arXiv.2001.05087

2020-01-15

Abstract:We present a general algorithm to order moves so as to speedup exact game solvers. It uses online learning of playout policies and Monte Carlo Tree Search. The learned policy and the information in the Monte Carlo tree are used to order moves in game solvers. They improve greatly the solving time for multiple games.

Artificial Intelligence
Combining Monte-Carlo Tree Search with Proof-Number Search

Elliot Doe,Mark H. M. Winands,Dennis J. N. J. Soemers,Cameron Browne

DOI: https://doi.org/10.48550/arXiv.2206.03965

2022-06-08

Abstract:Proof-Number Search (PNS) and Monte-Carlo Tree Search (MCTS) have been successfully applied for decision making in a range of games. This paper proposes a new approach called PN-MCTS that combines these two tree-search methods by incorporating the concept of proof and disproof numbers into the UCT formula of MCTS. Experimental results demonstrate that PN-MCTS outperforms basic MCTS in several games including Lines of Action, MiniShogi, Knightthrough, and Awari, achieving win rates up to 94.0%.

Artificial Intelligence
Lookahead Pathology in Monte-Carlo Tree Search

Khoi P. N. Nguyen,Raghuram Ramanujan

DOI: https://doi.org/10.1609/icaps.v34i1.31501

2024-06-08

Abstract:Monte-Carlo Tree Search (MCTS) is a search paradigm that first found prominence with its success in the domain of computer Go. Early theoretical work established the soundness and convergence bounds for Upper Confidence bounds applied to Trees (UCT), the most popular instantiation of MCTS; however, there remain notable gaps in our understanding of how UCT behaves in practice. In this work, we address one such gap by considering the question of whether UCT can exhibit lookahead pathology in adversarial settings -- a paradoxical phenomenon first observed in Minimax search where greater search effort leads to worse decision-making. We introduce a novel family of synthetic games that offer rich modeling possibilities while remaining amenable to mathematical analysis. Our theoretical and experimental results suggest that UCT is indeed susceptible to pathological behavior in a range of games drawn from this family.

Artificial Intelligence,Multiagent Systems
A Survey of Monte Carlo Tree Search Methods

Cameron B. Browne,Edward Powley,Daniel Whitehouse,Simon M. Lucas,Peter I. Cowling,Philipp Rohlfshagen,Stephen Tavener,Diego Perez,Spyridon Samothrakis,Simon Colton

DOI: https://doi.org/10.1109/tciaig.2012.2186810

2012-03-01

IEEE Transactions on Computational Intelligence and AI in Games

Abstract:Monte Carlo tree search (MCTS) is a recently proposed search method that combines the precision of tree search with the generality of random sampling. It has received considerable interest due to its spectacular success in the difficult problem of computer Go, but has also proved beneficial in a range of other domains. This paper is a survey of the literature to date, intended to provide a snapshot of the state of the art after the first five years of MCTS research. We outline the core algorithm's derivation, impart some structure on the many variations and enhancements that have been proposed, and summarize the results from the key game and nongame domains to which MCTS methods have been applied. A number of open research questions indicate that the field is ripe for future work.
Monte Carlo Search Algorithms Discovering Monte Carlo Tree Search Exploration Terms

Tristan Cazenave

2024-04-15

Abstract:Monte Carlo Tree Search and Monte Carlo Search have good results for many combinatorial problems. In this paper we propose to use Monte Carlo Search to design mathematical expressions that are used as exploration terms for Monte Carlo Tree Search algorithms. The optimized Monte Carlo Tree Search algorithms are PUCT and SHUSS. We automatically design the PUCT and the SHUSS root exploration terms. For small search budgets of 32 evaluations the discovered root exploration terms make both algorithms competitive with usual PUCT.

Artificial Intelligence
Monte Carlo Tree Search for a single target search game on a 2-D lattice

Elana Kozak,Scott Hottovy

DOI: https://doi.org/10.48550/arXiv.2011.14246

2020-11-29

Abstract:Monte Carlo Tree Search (MCTS) is a branch of stochastic modeling that utilizes decision trees for optimization, mostly applied to artificial intelligence (AI) game players. This project imagines a game in which an AI player searches for a stationary target within a 2-D lattice. We analyze its behavior with different target distributions and compare its efficiency to the Levy Flight Search, a model for animal foraging behavior. In addition to simulated data analysis we prove two theorems about the convergence of MCTS when computation constraints neglected.

Machine Learning
Monte Carlo Tree Search with Boltzmann Exploration

Michael Painter,Mohamed Baioumy,Nick Hawes,Bruno Lacerda

2024-04-11

Abstract:Monte-Carlo Tree Search (MCTS) methods, such as Upper Confidence Bound applied to Trees (UCT), are instrumental to automated planning techniques. However, UCT can be slow to explore an optimal action when it initially appears inferior to other actions. Maximum ENtropy Tree-Search (MENTS) incorporates the maximum entropy principle into an MCTS approach, utilising Boltzmann policies to sample actions, naturally encouraging more exploration. In this paper, we highlight a major limitation of MENTS: optimal actions for the maximum entropy objective do not necessarily correspond to optimal actions for the original objective. We introduce two algorithms, Boltzmann Tree Search (BTS) and Decaying ENtropy Tree-Search (DENTS), that address these limitations and preserve the benefits of Boltzmann policies, such as allowing actions to be sampled faster by using the Alias method. Our empirical analysis shows that our algorithms show consistent high performance across several benchmark domains, including the game of Go.

Artificial Intelligence,Machine Learning
Monte-Carlo Tree Search with Epsilon-Greedy for Game of Amazons

Chen Tian

DOI: https://doi.org/10.54254/2755-2721/6/20230956

2023-01-01

Abstract:The Game of the Amazons is an abstract strategy board game. It has a high computational complexity similar to the game of Go. Due to its NP-complete nature and large branching factor of game tree, finding the optimal move given a specific game state is infeasible and it is not trivial to design a computer algorithm that is competitive to an expert in the game of amazons. One way to tackle this problem is to leverage the Monte-Carlo Tree Search by using random simulations. In this article, a computationally cheap heuristic function is proposed and use together with Monte-Carlo Tree Search algorithm with Epsilon-Greedy policy aiming to design a competitive AI for the Game of the Amazon. The effectiveness of the -greedy based Monte-Carlo algorithm is compared to the widely used MCTS with Upper Confidence Bound and other classical tree search method such as breadth-first search, depth-first search, minmax search and alpha-beta pruning.
Playing With Monte-Carlo Tree Search [AI-eXplained]

Yunlong Zhao,Chengpeng Hu,Jialin Liu

DOI: https://doi.org/10.1109/mci.2023.3328150

IF: 9.809

2024-02-01

IEEE Computational Intelligence Magazine

Abstract:This paper provides an accessible explanation of the working mechanism of Monte-Carlo Tree Search, an influential search algorithm. The paper summarizes the procedure of Monte-Carlo Tree Search, including selection, expansion, simulation, and backpropagation. Additionally, immersive examples based on Tic-Tac-Toe, Go, and Sokoban, two two-player competitive games and a classic single-player puzzle game, are presented to illustrate how Monte-Carlo Tree Search works. The full article with interactive contents is published on IEEE Xplore.

computer science, artificial intelligence
Dual Monte Carlo Tree Search

Prashank Kadam,Ruiyang Xu,Karl Lieberherr

DOI: https://doi.org/10.48550/arXiv.2103.11517

2021-10-10

Abstract:AlphaZero, using a combination of Deep Neural Networks and Monte Carlo Tree Search (MCTS), has successfully trained reinforcement learning agents in a tabula-rasa way. The neural MCTS algorithm has been successful in finding near-optimal strategies for games through self-play. However, the AlphaZero algorithm has a significant drawback; it takes a long time to converge and requires high computational power due to complex neural networks for solving games like Chess, Go, Shogi, etc. Owing to this, it is very difficult to pursue neural MCTS research without cutting-edge hardware, which is a roadblock for many aspiring neural MCTS researchers. In this paper, we propose a new neural MCTS algorithm, called Dual MCTS, which helps overcome these drawbacks. Dual MCTS uses two different search trees, a single deep neural network, and a new update technique for the search trees using a combination of the PUCB, a sliding-window, and the epsilon-greedy algorithm. This technique is applicable to any MCTS based algorithm to reduce the number of updates to the tree. We show that Dual MCTS performs better than one of the most widely used neural MCTS algorithms, AlphaZero, for various symmetric and asymmetric games.

Artificial Intelligence,Multiagent Systems
Beyond games: a systematic review of neural Monte Carlo tree search applications

Marco Kemmerling,Daniel Lütticke,Robert H. Schmitt

DOI: https://doi.org/10.1007/s10489-023-05240-w

IF: 5.3

2023-12-28

Applied Intelligence

Abstract:Abstract The advent of AlphaGo and its successors marked the beginning of a new paradigm in playing games using artificial intelligence. This was achieved by combining Monte Carlo tree search, a planning procedure, and deep learning. While the impact on the domain of games has been undeniable, it is less clear how useful similar approaches are in applications beyond games and how they need to be adapted from the original methodology. We perform a systematic literature review of peer-reviewed articles detailing the application of neural Monte Carlo tree search methods in domains other than games. Our goal is to systematically assess how such methods are structured in practice and if their success can be extended to other domains. We find applications in a variety of domains, many distinct ways of guiding the tree search using learned policy and value functions, and various training methods. Our review maps the current landscape of algorithms in the family of neural monte carlo tree search as they are applied to practical problems, which is a first step towards a more principled way of designing such algorithms for specific problems and their requirements.

computer science, artificial intelligence
Monte Carlo Tree Search: a review of recent modifications and applications

Maciej Świechowski,Konrad Godlewski,Bartosz Sawicki,Jacek Mańdziuk

DOI: https://doi.org/10.1007/s10462-022-10228-y

IF: 9.588

2022-07-21

Artificial Intelligence Review

Abstract:Monte Carlo Tree Search (MCTS) is a powerful approach to designing game-playing bots or solving sequential decision problems. The method relies on intelligent tree search that balances exploration and exploitation. MCTS performs random sampling in the form of simulations and stores statistics of actions to make more educated choices in each subsequent iteration. The method has become a state-of-the-art technique for combinatorial games. However, in more complex games (e.g. those with a high branching factor or real-time ones) as well as in various practical domains (e.g. transportation, scheduling or security) an efficient MCTS application often requires its problem-dependent modification or integration with other techniques. Such domain-specific modifications and hybrid approaches are the main focus of this survey. The last major MCTS survey was published in 2012. Contributions that appeared since its release are of particular interest for this review.

computer science, artificial intelligence
Exploring search space trees using an adapted version of Monte Carlo tree search for combinatorial optimization problems

Jorik Jooken,Pieter Leyman,Tony Wauters,Patrick De Causmaecker

DOI: https://doi.org/10.1016/j.cor.2022.106070

2022-11-07

Abstract:In this article we propose a heuristic algorithm to explore search space trees associated with instances of combinatorial optimization problems. The algorithm is based on Monte Carlo tree search, a popular algorithm in game playing that is used to explore game trees and represents the state-of-the-art algorithm for a number of games. Several enhancements to Monte Carlo tree search are proposed that make the algorithm more suitable in a combinatorial optimization context. These enhancements exploit the combinatorial structure of the problem and aim to efficiently explore the search space tree by pruning subtrees, using a heuristic simulation policy, reducing the domains of variables by eliminating dominated value assignments and using a beam width. The algorithm was implemented with its components specifically tailored to two combinatorial optimization problems: the quay crane scheduling problem with non-crossing constraints and the 0-1 knapsack problem. For the first problem our algorithm surpasses the state-of-the-art results and several new best solutions are found for a benchmark set of instances. For the second problem our algorithm typically produces near-optimal solutions that are slightly worse than the state-of-the-art results, but it needs only a small fraction of the time to do so. These results indicate that the algorithm is competitive with the state-of-the-art for two entirely different combinatorial optimization problems.

computer science, interdisciplinary applications,engineering, industrial,operations research & management science
Playing Carcassonne with Monte Carlo Tree Search

Fred Valdez Ameneyro,Edgar Galvan,Anger Fernando Kuri Morales

DOI: https://doi.org/10.48550/arXiv.2009.12974

2020-10-05

Abstract:Monte Carlo Tree Search (MCTS) is a relatively new sampling method with multiple variants in the literature. They can be applied to a wide variety of challenging domains including board games, video games, and energy-based problems to mention a few. In this work, we explore the use of the vanilla MCTS and the MCTS with Rapid Action Value Estimation (MCTS-RAVE) in the game of Carcassonne, a stochastic game with a deceptive scoring system where limited research has been conducted. We compare the strengths of the MCTS-based methods with the Star2.5 algorithm, previously reported to yield competitive results in the game of Carcassonne when a domain-specific heuristic is used to evaluate the game states. We analyse the particularities of the strategies adopted by the algorithms when they share a common reward system. The MCTS-based methods consistently outperformed the Star2.5 algorithm given their ability to find and follow long-term strategies, with the vanilla MCTS exhibiting a more robust game-play than the MCTS-RAVE.

Artificial Intelligence
Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B

Di Zhang,Xiaoshui Huang,Dongzhan Zhou,Yuqiang Li,Wanli Ouyang

2024-06-13

Abstract:This paper introduces the MCT Self-Refine (MCTSr) algorithm, an innovative integration of Large Language Models (LLMs) with Monte Carlo Tree Search (MCTS), designed to enhance performance in complex mathematical reasoning tasks. Addressing the challenges of accuracy and reliability in LLMs, particularly in strategic and mathematical reasoning, MCTSr leverages systematic exploration and heuristic self-refine mechanisms to improve decision-making frameworks within LLMs. The algorithm constructs a Monte Carlo search tree through iterative processes of Selection, self-refine, self-evaluation, and Backpropagation, utilizing an improved Upper Confidence Bound (UCB) formula to optimize the exploration-exploitation balance. Extensive experiments demonstrate MCTSr's efficacy in solving Olympiad-level mathematical problems, significantly improving success rates across multiple datasets, including GSM8K, GSM Hard, MATH, and Olympiad-level benchmarks, including Math Odyssey, AIME, and OlympiadBench. The study advances the application of LLMs in complex reasoning tasks and sets a foundation for future AI integration, enhancing decision-making accuracy and reliability in LLM-driven applications.

Artificial Intelligence
Distributed Monte Carlo Tree Search: A Novel Technique and its Application to Computer Go

Lars Schaefers,Marco Platzner

DOI: https://doi.org/10.1109/tciaig.2014.2346997

2015-12-01

IEEE Transactions on Computational Intelligence and AI in Games

Abstract:Monte Carlo tree search (MCTS) has brought about great success regarding the evaluation of stochastic and deterministic games in recent years. We present and empirically analyze a data-driven parallelization approach for MCTS targeting large HPC clusters with Infiniband interconnect. Our implementation is based on OpenMPI and makes extensive use of its RDMA based asynchronous tiny message communication capabilities for effectively overlapping communication and computation. We integrate our parallel MCTS approach termed UCT-Treesplit in our state-of-the-art Go engine Gomorra and measure its strengths and limitations in a real-world setting. Our extensive experiments show that we can scale up to 128 compute nodes and 2048 cores in self-play experiments and, furthermore, give promising directions for additional improvement. The generality of our parallelization approach advocates its use to significantly improve the search quality of a huge number of current MCTS applications.
Towards a Characterisation of Monte-Carlo Tree Search Performance in Different Games

Dennis J.N.J. Soemers,Guillaume Bams,Max Persoon,Marco Rietjens,Dimitar Sladić,Stefan Stefanov,Kurt Driessens,Mark H.M. Winands

2024-06-13

Abstract:Many enhancements to Monte-Carlo Tree Search (MCTS) have been proposed over almost two decades of general game playing and other artificial intelligence research. However, our ability to characterise and understand which variants work well or poorly in which games is still lacking. This paper describes work on an initial dataset that we have built to make progress towards such an understanding: 268,386 plays among 61 different agents across 1494 distinct games. We describe a preliminary analysis and work on training predictive models on this dataset, as well as lessons learned and future plans for a new and improved version of the dataset.

Artificial Intelligence
Monte-Carlo Tree Search for Multi-Agent Pathfinding: Preliminary Results

Yelisey Pitanov,Alexey Skrynnik,Anton Andreychuk,Konstantin Yakovlev,Aleksandr Panov

2023-07-25

Abstract:In this work we study a well-known and challenging problem of Multi-agent Pathfinding, when a set of agents is confined to a graph, each agent is assigned a unique start and goal vertices and the task is to find a set of collision-free paths (one for each agent) such that each agent reaches its respective goal. We investigate how to utilize Monte-Carlo Tree Search (MCTS) to solve the problem. Although MCTS was shown to demonstrate superior performance in a wide range of problems like playing antagonistic games (e.g. Go, Chess etc.), discovering faster matrix multiplication algorithms etc., its application to the problem at hand was not well studied before. To this end we introduce an original variant of MCTS, tailored to multi-agent pathfinding. The crux of our approach is how the reward, that guides MCTS, is computed. Specifically, we use individual paths to assist the agents with the the goal-reaching behavior, while leaving them freedom to get off the track if it is needed to avoid collisions. We also use a dedicated decomposition technique to reduce the branching factor of the tree search procedure. Empirically we show that the suggested method outperforms the baseline planning algorithm that invokes heuristic search, e.g. A*, at each re-planning step.

Artificial Intelligence

The Mathematical Game

Proof Number Based Monte-Carlo Tree Search

Belief-state Monte-Carlo Tree Search for Phantom Games

Monte Carlo Game Solver

Combining Monte-Carlo Tree Search with Proof-Number Search

Lookahead Pathology in Monte-Carlo Tree Search

A Survey of Monte Carlo Tree Search Methods

Monte Carlo Search Algorithms Discovering Monte Carlo Tree Search Exploration Terms

Monte Carlo Tree Search for a single target search game on a 2-D lattice

Monte Carlo Tree Search with Boltzmann Exploration

Monte-Carlo Tree Search with Epsilon-Greedy for Game of Amazons

Playing With Monte-Carlo Tree Search [AI-eXplained]

Dual Monte Carlo Tree Search

Beyond games: a systematic review of neural Monte Carlo tree search applications

Monte Carlo Tree Search: a review of recent modifications and applications

Exploring search space trees using an adapted version of Monte Carlo tree search for combinatorial optimization problems

Playing Carcassonne with Monte Carlo Tree Search

Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B

Distributed Monte Carlo Tree Search: A Novel Technique and its Application to Computer Go

Towards a Characterisation of Monte-Carlo Tree Search Performance in Different Games

Monte-Carlo Tree Search for Multi-Agent Pathfinding: Preliminary Results