Abstract:In recent years, reinforcement learning has seen interest because of deep Q-Learning, where the model is a convolutional neural network. Deep Q-Learning has shown promising results in games such as Atari and AlphaGo. Instead of learning the entire Q-table, it learns an estimate of the Q function that determines a state's policy action. We use Q-Learning and deep Q-learning, to learn control policies of four constraint satisfaction games (15-Puzzle, Minesweeper, 2048, and Sudoku). 15-Puzzle is a sliding permutation puzzle and provides a challenge in addressing its large state space. Minesweeper and Sudoku involve partially observable states and guessing. 2048 is also a sliding puzzle but allows for easier state representation (compared to 15-Puzzle) and uses interesting reward shaping to solve the game. These games offer unique insights into the potential and limits of reinforcement learning. The Q agent is trained with no rules of the game, with only the reward corresponding to each state's action. Our unique contribution is in choosing the reward structure, state representation, and formulation of the deep neural network. For low shuffle, 15-Puzzle, achieves a 100% win rate, the medium and high shuffle achieve about 43% and 22% win rates respectively. On a standard 16x16 Minesweeper board, both low and high-density boards achieve close to 45% win rate, whereas medium density boards have a low win rate of 15%. For 2048, the 1024 win rate was achieved with significant ease (100%) with high win rates for 2048, 4096, 8192 and 16384 as 40%, 0.05%, 0.01% and 0.004% , respectively. The easy Sudoku games had a win rate of 7%, while medium and hard games had 2.1% and 1.2% win rates, respectively. This paper explores the environment complexity and behavior of a subset of constraint games using reward structures which can get us closer to understanding how humans learn.

Hierarchical Deep Reinforcement Learning Agent with Counter Self-play on Competitive Games

Self-play Reinforcement Learning with Comprehensive Critic in Computer Games

Hierarchical Reinforcement Learning for Multi-agent MOBA Game

From mimic to counteract: a two-stage reinforcement learning algorithm for Google research football

Efficient Competitive Self-Play Policy Optimization

Cooperative multi-agent target searching: a deep reinforcement learning approach based on parallel hindsight experience replay

On Multi-Agent Learning in Team Sports Games

A Comparison of Self-Play Algorithms Under a Generalized Framework

Minimax Exploiter: A Data Efficient Approach for Competitive Self-Play

Towards Playing Full MOBA Games with Deep Reinforcement Learning

Hierarchical Method for Cooperative Multiagent Reinforcement Learning in Markov Decision Processes

Design of Artificial Intelligence Agents for Games using Deep Reinforcement Learning

Learning to Play Air Hockey with Model-Based Deep Reinforcement Learning

Learn a Robust Policy in Adversarial Games Via Playing with an Expert Opponent

Natural Language-Based Human–Machine Collaborative Learning Games Algorithm Based on Deep Rein-Forcement Learning

Enhanced Rolling Horizon Evolution Algorithm with Opponent Model Learning: Results for the Fighting Game AI Competition

Deep reinforcement learning algorithm based on multi-agent parallelism and its application in game environment

Neural Auto-Curricula

Cooperative multi-agent game based on reinforcement learning

Provable Self-Play Algorithms for Competitive Reinforcement Learning

Reinforcement Learning For Constraint Satisfaction Game Agents (15-Puzzle, Minesweeper, 2048, and Sudoku)