Abstract:Reinforcement learning (RL) trains agents to accomplish complex tasks through environmental interaction data, but its capacity is also limited by the scope of the available data. To obtain a knowledgeable agent, a promising approach is to leverage the knowledge from large language models (LLMs). Despite previous studies combining LLMs with RL, seamless integration of the two components remains challenging due to their semantic gap. This paper introduces a novel method, Knowledgeable Agents from Language Model Rollouts (KALM), which extracts knowledge from LLMs in the form of imaginary rollouts that can be easily learned by the agent through offline reinforcement learning methods. The primary challenge of KALM lies in LLM grounding, as LLMs are inherently limited to textual data, whereas environmental data often comprise numerical vectors unseen to LLMs. To address this, KALM fine-tunes the LLM to perform various tasks based on environmental data, including bidirectional translation between natural language descriptions of skills and their corresponding rollout data. This grounding process enhances the LLM's comprehension of environmental dynamics, enabling it to generate diverse and meaningful imaginary rollouts that reflect novel skills. Initial empirical evaluations on the CLEVR-Robot environment demonstrate that KALM enables agents to complete complex rephrasings of task goals and extend their capabilities to novel tasks requiring unprecedented optimal behaviors. KALM achieves a success rate of 46% in executing tasks with unseen goals, substantially surpassing the 26% success rate achieved by baseline methods. Furthermore, KALM effectively enables the LLM to comprehend environmental dynamics, resulting in the generation of meaningful imaginary rollouts that reflect novel skills and demonstrate the seamless integration of large language models and reinforcement learning.

Knowledge-aware Leap-LSTM: Integrating Prior Knowledge into Leap-LSTM towards Faster Long Text Classification

Leap-LSTM: Enhancing Long Short-Term Memory for Text Categorization

Novel Efficient RNN and LSTM-Like Architectures: Recurrent and Gated Broad Learning Systems and Their Applications for Text Classification

Long short-term memory (LSTM)-based news classification model

Learning to Skim Text

FastLearn: A Rapid Learning Agent for Chat Models to Acquire Latest Knowledge

Attention-based BiLSTM fused CNN with gating mechanism model for Chinese long text classification

Knowledge Efficient Deep Learning for Natural Language Processing

Recognizing Textual Entailment via Multi-task Knowledge Assisted LSTM.

Local Bidirectional Long Short Term Memory for Text Classification

A C-LSTM Neural Network for Text Classification

Convolutional Long Short-term Memory for Long Length Document Classification

Integrating LSTM and BERT for Long-Sequence Data Analysis in Intelligent Tutoring Systems

Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs

Evolving Long Short-Term Memory Network-Based Text Classification

Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs

BKT-LSTM: Efficient Student Modeling for knowledge tracing and student performance prediction

Research on the application of knowledge mapping and knowledge structure construction based on adaptive learning model

NEWLSTM: an Optimized Long Short-Term Memory Language Model for Sequence Prediction.

Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts

Extreme-Long-short Term Memory for Time-series Prediction