Abstract:Deep reinforcement learning (RL) provides powerful methods for training optimal sequential decision-making agents. As collecting real-world interactions can entail additional costs and safety risks, the common paradigm of sim2real conducts training in a simulator, followed by real-world deployment. Unfortunately, RL agents easily overfit to the choice of simulated training environments, and worse still, learning ends when the agent masters the specific set of simulated environments. In contrast, the real world is highly open-ended, featuring endlessly evolving environments and challenges, making such RL approaches unsuitable. Simply randomizing over simulated environments is insufficient, as it requires making arbitrary distributional assumptions and can be combinatorially less likely to sample specific environment instances that are useful for learning. An ideal learning process should automatically adapt the training environment to maximize the learning potential of the agent over an open-ended task space that matches or surpasses the complexity of the real world. This thesis develops a class of methods called Unsupervised Environment Design (UED), which aim to produce such open-ended processes. Given an environment design space, UED automatically generates an infinite sequence or curriculum of training environments at the frontier of the learning agent's capabilities. Through extensive empirical studies and theoretical arguments founded on minimax-regret decision theory and game theory, the findings in this thesis show that UED autocurricula can produce RL agents exhibiting significantly improved robustness and generalization to previously unseen environment instances. Such autocurricula are promising paths toward open-ended learning systems that achieve more general intelligence by continually generating and mastering additional challenges of their own design.

Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks

XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX

CoRL: Environment Creation and Management Focused on System Integration

Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAX

NovGrid: A Flexible Grid World for Evaluating Agent Response to Novelty

GriddlyJS: A Web IDE for Reinforcement Learning

AI Safety Gridworlds

Discovering Minimal Reinforcement Learning Environments

NAVIX: Scaling MiniGrid Environments with JAX

minimax: Efficient Baselines for Autocurricula in JAX

Craftium: An Extensible Framework for Creating Reinforcement Learning Environments

Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban

IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agents

Mini Honor of Kings: A Lightweight Environment for Multi-Agent Reinforcement Learning

Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms

Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI

GenRL: Multimodal-foundation world models for generalization in embodied agents

CaiRL: A High-Performance Reinforcement Learning Environment Toolkit

GridToPix: Training Embodied Agents with Minimal Supervision

NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds

Learning Curricula in Open-Ended Worlds