Abstract:Recent SOTA approaches for embodied learning via interaction directly employ large language models (LLMs) as agents to determine the next steps in an environment. Due to their world knowledge and reasoning capabilities, LLM agents achieve stronger performance than previous smaller agents based on reinforcement learning (RL); however, frequently calling LLMs is slow and expensive. Instead of directly employing LLMs as agents, can we use LLMs' reasoning capabilities to adaptively create training environments to help smaller RL agents learn useful skills that they are weak at? We propose EnvGen, a novel framework to address this question. We first prompt an LLM to generate training environments by giving it the task description and simulator objectives that the agents should learn and then asking it to generate a set of environment configurations (e.g., different terrains, items initially given to agents, etc.). Next, we train a small RL agent in a mixture of the original and LLM-generated environments. Then, we enable the LLM to continuously adapt the generated environments to progressively improve the skills that the agent is weak at, by providing feedback to the LLM in the form of the agent's performance. We demonstrate the usefulness of EnvGen with comprehensive experiments in Crafter and Heist environments. We find that a small RL agent trained with EnvGen can outperform SOTA methods, including a GPT-4 agent, and learns long-horizon tasks significantly faster. We also show that using an LLM to adapt environments dynamically outperforms curriculum learning approaches and how the environments are adapted to help improve RL agents' weaker skills over time. Additionally, EnvGen is substantially more efficient as it only uses a small number of LLM calls (e.g., 4 in total), whereas LLM agents require thousands of calls. Lastly, we present detailed ablation studies for EnvGen design choices.

Adversarial Reinforcement Learning for Procedural Content Generation

PCGRL: Procedural Content Generation via Reinforcement Learning

PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generators

Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments

G-PCGRL: Procedural Graph Data Generation via Reinforcement Learning

Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design

Adversarial Environment Generation for Learning to Navigate the Web

Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning

Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewards

Learning-Based Procedural Content Generation

Programmatic Modeling and Generation of Real-Time Strategic Soccer Environments for Reinforcement Learning

Procedural Content Generation via Machine Learning (PCGML)

Generating Automatic Curricula via Self-Supervised Active Domain Randomization

Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning

Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments

Stabilizing Unsupervised Environment Design with a Learned Adversary

Deep Surrogate Assisted Generation of Environments

RLSS: A Deep Reinforcement Learning Algorithm for Sequential Scene Generation

Adversarial Environment Design via Regret-Guided Diffusion Models

Negatively Correlated Ensemble Reinforcement Learning for Online Diverse Game Level Generation

EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents