Abstract:There has been a growing interest in developing learner models to enhance learning and teaching experiences in educational environments. However, existing works have primarily focused on structured environments relying on meticulously crafted representations of tasks, thereby limiting the agent's ability to generalize skills across tasks. In this paper, we aim to enhance the generalization capabilities of agents in open-ended text-based learning environments by integrating Reinforcement Learning (RL) with Large Language Models (LLMs). We investigate three types of agents: (i) RL-based agents that utilize natural language for state and action representations to find the best interaction strategy, (ii) LLM-based agents that leverage the model's general knowledge and reasoning through prompting, and (iii) hybrid LLM-assisted RL agents that combine these two strategies to improve agents' performance and generalization. To support the development and evaluation of these agents, we introduce PharmaSimText, a novel benchmark derived from the PharmaSim virtual pharmacy environment designed for practicing diagnostic conversations. Our results show that RL-based agents excel in task completion but lack in asking quality diagnostic questions. In contrast, LLM-based agents perform better in asking diagnostic questions but fall short of completing the task. Finally, hybrid LLM-assisted RL agents enable us to overcome these limitations, highlighting the potential of combining RL and LLMs to develop high-performing agents for open-ended learning environments.

RTFM: Generalising to Novel Environment Dynamics via Reading

Deep Reinforcement Learning for NLP.

Feudal Reinforcement Learning by Reading Manuals

Ask Before You Act: Generalising to Novel Environments by Asking Questions

Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals

GenRL: Multimodal-foundation world models for generalization in embodied agents

EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning

Using reinforcement learning to learn how to play text-based games

Grounding Language for Transfer in Deep Reinforcement Learning

Natural Language Reinforcement Learning

Generalization in Text-based Games via Hierarchical Reinforcement Learning

Learning to Model the World with Language

Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs

Learning Invariable Semantical Representation from Language for Extensible Policy Generalization

Learning to Generalize for Sequential Decision Making

Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

Learning Parsimonious Dynamics for Generalization in Reinforcement Learning

Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models

Language Understanding for Text-based Games Using Deep Reinforcement Learning

Natural Language Specification of Reinforcement Learning Policies Through Differentiable Decision Trees

ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Models