Abstract:In recent years, reinforcement learning has seen interest because of deep Q-Learning, where the model is a convolutional neural network. Deep Q-Learning has shown promising results in games such as Atari and AlphaGo. Instead of learning the entire Q-table, it learns an estimate of the Q function that determines a state's policy action. We use Q-Learning and deep Q-learning, to learn control policies of four constraint satisfaction games (15-Puzzle, Minesweeper, 2048, and Sudoku). 15-Puzzle is a sliding permutation puzzle and provides a challenge in addressing its large state space. Minesweeper and Sudoku involve partially observable states and guessing. 2048 is also a sliding puzzle but allows for easier state representation (compared to 15-Puzzle) and uses interesting reward shaping to solve the game. These games offer unique insights into the potential and limits of reinforcement learning. The Q agent is trained with no rules of the game, with only the reward corresponding to each state's action. Our unique contribution is in choosing the reward structure, state representation, and formulation of the deep neural network. For low shuffle, 15-Puzzle, achieves a 100% win rate, the medium and high shuffle achieve about 43% and 22% win rates respectively. On a standard 16x16 Minesweeper board, both low and high-density boards achieve close to 45% win rate, whereas medium density boards have a low win rate of 15%. For 2048, the 1024 win rate was achieved with significant ease (100%) with high win rates for 2048, 4096, 8192 and 16384 as 40%, 0.05%, 0.01% and 0.004% , respectively. The easy Sudoku games had a win rate of 7%, while medium and hard games had 2.1% and 1.2% win rates, respectively. This paper explores the environment complexity and behavior of a subset of constraint games using reward structures which can get us closer to understanding how humans learn.

What makes math problems hard for reinforcement learning: a case study

Turning Mathematics Problems into Games: Reinforcement Learning and Gröbner bases together solve Integer Feasibility Problems

MathDQN: Solving Arithmetic Word Problems Via Deep Reinforcement Learning.

Reinforcement Learning For Constraint Satisfaction Game Agents (15-Puzzle, Minesweeper, 2048, and Sudoku)

A Reinforcement Learning Environment for Mathematical Reasoning via Program Synthesis

AlphaSnake: Policy Iteration on a Nondeterministic NP-hard Markov Decision Process (student Abstract)

Learning a Prior for Monte Carlo Search by Replaying Solutions to Combinatorial Problems

AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling

Hardness in Markov Decision Processes: Theory and Practice

A Novel Automated Curriculum Strategy to Solve Hard Sokoban Planning Instances

AlphaSnake: Policy Iteration on a Nondeterministic NP-hard Markov Decision Process

Neural Math Word Problem Solver with Reinforcement Learning.

Dealing with Sparse Rewards in Reinforcement Learning

Solving for X and Beyond: Can Large Language Models Solve Complex Math Problems with More-Than-Two Unknowns?

Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learning

Problem Solving at the Edge of Chaos: Entropy, Puzzles and the Sudoku Freezing Transition

Is Long Horizon Reinforcement Learning More Difficult Than Short Horizon Reinforcement Learning?

Large-Scale Study of Curiosity-Driven Learning

Analysing Mathematical Reasoning Abilities of Neural Models

HARP: A challenging human-annotated math reasoning benchmark

PUZZLES: A Benchmark for Neural Algorithmic Reasoning