Abstract:Deep reinforcement learning (RL) provides powerful methods for training optimal sequential decision-making agents. As collecting real-world interactions can entail additional costs and safety risks, the common paradigm of sim2real conducts training in a simulator, followed by real-world deployment. Unfortunately, RL agents easily overfit to the choice of simulated training environments, and worse still, learning ends when the agent masters the specific set of simulated environments. In contrast, the real world is highly open-ended, featuring endlessly evolving environments and challenges, making such RL approaches unsuitable. Simply randomizing over simulated environments is insufficient, as it requires making arbitrary distributional assumptions and can be combinatorially less likely to sample specific environment instances that are useful for learning. An ideal learning process should automatically adapt the training environment to maximize the learning potential of the agent over an open-ended task space that matches or surpasses the complexity of the real world. This thesis develops a class of methods called Unsupervised Environment Design (UED), which aim to produce such open-ended processes. Given an environment design space, UED automatically generates an infinite sequence or curriculum of training environments at the frontier of the learning agent's capabilities. Through extensive empirical studies and theoretical arguments founded on minimax-regret decision theory and game theory, the findings in this thesis show that UED autocurricula can produce RL agents exhibiting significantly improved robustness and generalization to previously unseen environment instances. Such autocurricula are promising paths toward open-ended learning systems that achieve more general intelligence by continually generating and mastering additional challenges of their own design.

Learning the Optimal Power Flow: Environment Design Matters

Combining Reinforcement Learning with Mathematical Programming: an Approach for Optimal Design of Heat Exchanger Networks

Physics-Informed Reinforcement Learning for Real-Time Optimal Power Flow with Renewable Energy Resources

Importance of Environment Design in Reinforcement Learning: A Study of a Robotic Environment

A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning

Safe deep reinforcement learning for real-time AC optimal power flow: A near-optimal solution

A Deep Reinforcement Learning Based Approach for Optimal Active Power Dispatch

Can Learned Optimization Make Reinforcement Learning Less Difficult?

Optimizing Load Scheduling in Power Grids Using Reinforcement Learning and Markov Decision Processes

Learning Curricula in Open-Ended Worlds

Learning to Design Games: Strategic Environments in Reinforcement Learning

A Safe DRL Method for Fast Solution of Real-Time Optimal Power Flow

Optimal Operable Power Flow: Sample-efficient Holomorphic Embedding-based Reinforcement Learning

Design Process is a Reinforcement Learning Problem

RLOR: A Flexible Framework of Deep Reinforcement Learning for Operation Research

Optimal Power Flow in a highly renewable power system based on attention neural networks

A General Real-time OPF Algorithm Using DDPG with Multiple Simulation Platforms

Automatic Environment Shaping is the Next Frontier in RL

Real-time Power Scheduling Through Reinforcement Learning from Demonstrations

Environment Design for Inverse Reinforcement Learning

Reinforcement Learning for Efficient Power Systems Planning: A Review of Operational and Expansion Strategies