Abstract:Reinforcement learning (RL) trains agents to accomplish complex tasks through environmental interaction data, but its capacity is also limited by the scope of the available data. To obtain a knowledgeable agent, a promising approach is to leverage the knowledge from large language models (LLMs). Despite previous studies combining LLMs with RL, seamless integration of the two components remains challenging due to their semantic gap. This paper introduces a novel method, Knowledgeable Agents from Language Model Rollouts (KALM), which extracts knowledge from LLMs in the form of imaginary rollouts that can be easily learned by the agent through offline reinforcement learning methods. The primary challenge of KALM lies in LLM grounding, as LLMs are inherently limited to textual data, whereas environmental data often comprise numerical vectors unseen to LLMs. To address this, KALM fine-tunes the LLM to perform various tasks based on environmental data, including bidirectional translation between natural language descriptions of skills and their corresponding rollout data. This grounding process enhances the LLM's comprehension of environmental dynamics, enabling it to generate diverse and meaningful imaginary rollouts that reflect novel skills. Initial empirical evaluations on the CLEVR-Robot environment demonstrate that KALM enables agents to complete complex rephrasings of task goals and extend their capabilities to novel tasks requiring unprecedented optimal behaviors. KALM achieves a success rate of 46% in executing tasks with unseen goals, substantially surpassing the 26% success rate achieved by baseline methods. Furthermore, KALM effectively enables the LLM to comprehend environmental dynamics, resulting in the generation of meaningful imaginary rollouts that reflect novel skills and demonstrate the seamless integration of large language models and reinforcement learning.

Guiding Pretraining in Reinforcement Learning with Large Language Models

Efficient Reinforcement Learning with Large Language Model Priors

Learning Reward for Robot Skills Using Large Language Models via Self-Alignment

On the Modeling Capabilities of Large Language Models for Sequential Decision Making

World Models with Hints of Large Language Models for Goal Achieving

Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Models

From Laws to Motivation: Guiding Exploration through Law-Based Reasoning and Rewards

Mental Modeling of Reinforcement Learning Agents by Language Models

Words as Beacons: Guiding RL Agents with High-Level Language Prompts

Integrating Large Language Models and Reinforcement Learning for Non-Linear Reasoning

Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models

Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting

Teaching Large Language Models to Reason with Reinforcement Learning

Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models

LGR2: Language Guided Reward Relabeling for Accelerating Hierarchical Reinforcement Learning

Language Guided Exploration for RL Agents in Text Environments