Abstract:Deep reinforcement learning (RL) provides powerful methods for training optimal sequential decision-making agents. As collecting real-world interactions can entail additional costs and safety risks, the common paradigm of sim2real conducts training in a simulator, followed by real-world deployment. Unfortunately, RL agents easily overfit to the choice of simulated training environments, and worse still, learning ends when the agent masters the specific set of simulated environments. In contrast, the real world is highly open-ended, featuring endlessly evolving environments and challenges, making such RL approaches unsuitable. Simply randomizing over simulated environments is insufficient, as it requires making arbitrary distributional assumptions and can be combinatorially less likely to sample specific environment instances that are useful for learning. An ideal learning process should automatically adapt the training environment to maximize the learning potential of the agent over an open-ended task space that matches or surpasses the complexity of the real world. This thesis develops a class of methods called Unsupervised Environment Design (UED), which aim to produce such open-ended processes. Given an environment design space, UED automatically generates an infinite sequence or curriculum of training environments at the frontier of the learning agent's capabilities. Through extensive empirical studies and theoretical arguments founded on minimax-regret decision theory and game theory, the findings in this thesis show that UED autocurricula can produce RL agents exhibiting significantly improved robustness and generalization to previously unseen environment instances. Such autocurricula are promising paths toward open-ended learning systems that achieve more general intelligence by continually generating and mastering additional challenges of their own design.

OPEn: An Open-ended Physics Environment for Learning Without a Task

Learning Curricula in Open-Ended Worlds

OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research

Emergence of Collective Open-Ended Exploration from Decentralized Meta-Reinforcement Learning

Open Problem: Active Representation Learning

Open-Ended Learning Strategies for Learning Complex Locomotion Skills

OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code

OPEB: Open Physical Environment Benchmark for Artificial Intelligence

Learning to explore by reinforcement over high-level options

Stabilizing Unsupervised Environment Design with a Learned Adversary

NavRep: Unsupervised Representations for Reinforcement Learning of Robot Navigation in Dynamic Human Environments

OMNI: Open-endedness via Models of human Notions of Interestingness

Curiosity & Entropy Driven Unsupervised RL in Multiple Environments

MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning

Object-centric Representations for Interactive Online Learning with Non-Parametric Methods

OMPO: A Unified Framework for RL under Policy and Dynamics Shifts

Open-World Reinforcement Learning over Long Short-Term Imagination

REAL-X -- Robot open-Ended Autonomous Learning Architectures: Achieving Truly End-to-End Sensorimotor Autonomous Learning Systems

Probing Physics Knowledge Using Tools from Developmental Psychology

Learning Exploration Policies for Navigation

Interesting Object, Curious Agent: Learning Task-Agnostic Exploration