Abstract:Recent advancements in large language models (LLMs) have enabled understanding webpage contexts, product details, and human instructions. Utilizing LLMs as the foundational architecture for either reward models or policies in reinforcement learning has gained popularity -- a notable achievement is the success of InstructGPT. RL algorithms have been instrumental in maximizing long-term customer satisfaction and avoiding short-term, myopic goals in industrial recommender systems, which often rely on deep learning models to predict immediate clicks or purchases. In this project, several RL methods are implemented and evaluated using the WebShop benchmark environment, data, simulator, and pre-trained model checkpoints. The goal is to train an RL agent to maximize the purchase reward given a detailed human instruction describing a desired product. The RL agents are developed by fine-tuning a pre-trained BERT model with various objectives, learning from preferences without a reward model, and employing contemporary training techniques such as Proximal Policy Optimization (PPO) as used in InstructGPT, and Direct Preference Optimization (DPO). This report also evaluates the RL agents trained using generative trajectories. Evaluations were conducted using Thompson sampling in the WebShop simulator environment. The simulated online experiments demonstrate that agents trained on generated trajectories exhibited comparable task performance to those trained using human trajectories. This has demonstrated an example of an extremely low-cost data-efficient way of training reinforcement learning agents. Also, with limited training time (<2hours), without utilizing any images, a DPO agent achieved a 19% success rate after approximately 3000 steps or 30 minutes of training on T4 GPUs, compared to a PPO agent, which reached a 15% success rate.

Interactive Narrative Personalization with Deep Reinforcement Learning

Deep Reinforcement Learning for Dialogue Generation

Deep Reinforcement Learning for NLP.

High-Fidelity Simulated Players for Interactive Narrative Planning

Investigating deep reinforcement learning techniques in personalized dialogue generation

Session-based Interactive Recommendation Via Deep Reinforcement Learning

Reinforcement learning for game personalization on edge devices

Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations

Dialogue Shaping: Empowering Agents through NPC Interaction

Deep Policy Networks for NPC Behaviors that Adapt to Changing Design Parameters in Roguelike Games

NarrativePlay: Interactive Narrative Understanding

Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning.

Situated Language Learning via Interactive Narratives

Deep Reinforcement Learning-Driven Smart and Dynamic Mass Personalization

Learning to Play Text-based Adventure Games with Maximum Entropy Reinforcement Learning

NarrativePlay: An Automated System for Crafting Visual Worlds in Novels for Role-Playing

An Extremely Data-efficient and Generative LLM-based Reinforcement Learning Agent for Recommenders

Reward Constrained Interactive Recommendation with Natural Language Feedback

Goal-Directed Story Generation: Augmenting Generative Language Models with Reinforcement Learning

STARLING: Self-supervised Training of Text-based Reinforcement Learning Agent with Large Language Models