Abstract:We introduce gym-city, a Reinforcement Learning environment that uses SimCity 1's game engine to simulate an urban environment, wherein agents might seek to optimize one or a combination of any number of city-wide metrics, on gameboards of various sizes. We focus on population, and analyze our agents' ability to generalize to larger map-sizes than those seen during training. The environment is interactive, allowing a human player to build alongside agents during training and inference, potentially influencing the course of their learning, or manually probing and evaluating their performance. To test our agents' ability to capture distance-agnostic relationships between elements of the gameboard, we design a minigame within the environment which is, by design, unsolvable at large enough scales given strictly local strategies. Given the game engine's extensive use of Cellular Automata, we also train our agents to "play" Conway's Game of Life -- again optimizing for population -- and examine their behaviour at multiple scales. To make our models compatible with variable-scale gameplay, we use Neural Networks with recursive weights and structure -- fractals to be truncated at different depths, dependent upon the size of the gameboard.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the following two main problems: 1. **Generalization ability of reinforcement learning agents in city simulations of different scales**: - The paper introduces a reinforcement learning environment named `gym - city`, which uses the game engine of "SimCity 1" to simulate the urban environment. In this environment, the agent's goal is to optimize certain global indicators of the city (such as population). The focus of the research is to evaluate whether these agents can generalize the strategies learned during training to maps of a larger scale than those used during training. - Specifically, the paper explores whether agents can capture the distance - independent relationships between game board elements and maintain effective performance on larger - scale maps. 2. **Achieving multi - scale gameplay using fractal neural networks**: - In order to enable the model to adapt to different - scale gameplay, the author introduces neural networks with recursive weights and structures - namely, fractal neural networks (Fractal Neural Networks). These networks can truncate the fractal structure at different depths according to the size of the game board, thus ensuring the effectiveness of the model at different scales. - The paper also explores the performance of fractal neural networks when handling tasks that require global strategies (such as connecting distant areas), especially in a mini - game designed where the agent must connect a far - away residential area and a nuclear power plant. In addition, the paper also involves the following aspects of problems: - **The influence of human player interaction**: In the study, human players are allowed to interact with agents during training and inference to observe how this interaction affects the agent's learning process and performance. - **The application of cellular automata**: Since the "SimCity 1" game engine makes extensive use of cellular automata (Cellular Automata), the paper also explores the performance of agents in "Conway's Game of Life", with the goal of optimizing the population, to further examine their behavior at different scales. In summary, the core problem of this paper is to explore the generalization ability and adaptability of reinforcement learning agents in urban simulation tasks, especially their performance on maps of different scales, and to improve the performance of agents in global - strategy tasks by introducing fractal neural networks.

Using Fractal Neural Networks to Play SimCity 1 and Conway's Game of Life at Variable Scales

Playing a 2D Game Indefinitely using NEAT and Reinforcement Learning

Monte Carlo Neural Fictitious Self-Play: Achieve Approximate Nash equilibrium of Imperfect-Information Games.

Train on Small, Play the Large: Scaling Up Board Games with AlphaZero and GNN

PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generators

Cellular Automata and Fractal Urban Form: A Cellular Modelling Approach to the Evolution of Urban Land-Use Patterns

Scaling Laws for Imitation Learning in Single-Agent Games

Game of Intelligent Life

Optimizing Transportation Dynamics at a City-Scale Using a Reinforcement Learning Framework

Urban Planning Games and Simulations: From Board Games to Artificial Environments

Reinforcement Learning For Constraint Satisfaction Game Agents (15-Puzzle, Minesweeper, 2048, and Sudoku)

Simulation of Compact Cities based on the Integration of Cellular Automata and GIS

Emergent Solutions to High-Dimensional Multitask Reinforcement Learning

On equilibrium Metropolis simulations on self-organized urban street networks

Optimality-based Analysis of XCSF Compaction in Discrete Reinforcement Learning

Neural Auto-Curricula

Programmatic Modeling and Generation of Real-Time Strategic Soccer Environments for Reinforcement Learning

Turning Mathematics Problems into Games: Reinforcement Learning and Gröbner bases together solve Integer Feasibility Problems

Self-organizing Networks of Cities: Scaling Laws, Symmetry, Complexity and Optimization

Utilizing Multi Agent Simulation to Explore Cities

Playing Tetris with Reinforcement Learning