Abstract:Recent SOTA approaches for embodied learning via interaction directly employ large language models (LLMs) as agents to determine the next steps in an environment. Due to their world knowledge and reasoning capabilities, LLM agents achieve stronger performance than previous smaller agents based on reinforcement learning (RL); however, frequently calling LLMs is slow and expensive. Instead of directly employing LLMs as agents, can we use LLMs' reasoning capabilities to adaptively create training environments to help smaller RL agents learn useful skills that they are weak at? We propose EnvGen, a novel framework to address this question. We first prompt an LLM to generate training environments by giving it the task description and simulator objectives that the agents should learn and then asking it to generate a set of environment configurations (e.g., different terrains, items initially given to agents, etc.). Next, we train a small RL agent in a mixture of the original and LLM-generated environments. Then, we enable the LLM to continuously adapt the generated environments to progressively improve the skills that the agent is weak at, by providing feedback to the LLM in the form of the agent's performance. We demonstrate the usefulness of EnvGen with comprehensive experiments in Crafter and Heist environments. We find that a small RL agent trained with EnvGen can outperform SOTA methods, including a GPT-4 agent, and learns long-horizon tasks significantly faster. We also show that using an LLM to adapt environments dynamically outperforms curriculum learning approaches and how the environments are adapted to help improve RL agents' weaker skills over time. Additionally, EnvGen is substantially more efficient as it only uses a small number of LLM calls (e.g., 4 in total), whereas LLM agents require thousands of calls. Lastly, we present detailed ablation studies for EnvGen design choices.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to utilize the reasoning ability of large - language models (LLMs) to generate and adaptively adjust the training environment, thereby helping small reinforcement learning (RL) agents learn complex skills more effectively, especially those long - cycle tasks. In traditional methods, directly using LLMs as agents for decision - making, although having strong performance, is inefficient and costly. In contrast, the method EnvGen proposed in this paper generates diverse training environments by only invoking LLMs at critical stages and dynamically adjusts these environments according to the performance feedback of the agents to specifically improve the agents' performance on specific skills while maintaining an efficient training process. This method not only improves the learning efficiency of the agents but also significantly reduces the training cost.

EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents

AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation

Enabling Efficient Interaction between an Algorithm Agent and an LLM: A Reinforcement Learning Approach

Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

Learning Generative Interactive Environments By Trained Agent Exploration

Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

Words as Beacons: Guiding RL Agents with High-Level Language Prompts

Grounding Large Language Models In Embodied Environment With Imperfect World Models

AgentGym: Evolving Large Language Model-based Agents across Diverse Environments

LLM Augmented Hierarchical Agents

LLM-POET: Evolving Complex Environments using Large Language Models

LEGENT: Open Platform for Embodied Agents

GenRL: Multimodal-foundation world models for generalization in embodied agents

DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback

ExpeL: LLM Agents Are Experiential Learners

Adversarial Environment Generation for Learning to Navigate the Web

Empowering Large Language Model Agents through Action Learning

EnvBridge: Bridging Diverse Environments with Cross-Environment Knowledge Transfer for Embodied AI

Leveraging Environment Interaction for Automated PDDL Translation and Planning with Large Language Models