Abstract:Large Language Models (LLMs) have achieved remarkable success at tasks like summarization that involve a single turn of interaction. However, they can still struggle with multi-turn tasks like dialogue that require long-term planning. Previous works on multi-turn dialogue extend single-turn reinforcement learning from human feedback (RLHF) methods to the multi-turn setting by treating all prior dialogue turns as a long context. Such approaches suffer from covariate shift: the conversations in the training set have previous turns generated by some reference policy, which means that low training error may not necessarily correspond to good performance when the learner is actually in the conversation loop. In response, we introduce REgressing the RELative FUture (REFUEL), an efficient policy optimization approach designed to address multi-turn RLHF in LLMs. REFUEL employs a single model to estimate $Q$-values and trains on self-generated data, addressing the covariate shift issue. REFUEL frames the multi-turn RLHF problem as a sequence of regression tasks on iteratively collected datasets, enabling ease of implementation. Theoretically, we prove that REFUEL can match the performance of any policy covered by the training set. Empirically, we evaluate our algorithm by using Llama-3.1-70B-it to simulate a user in conversation with our model. REFUEL consistently outperforms state-of-the-art methods such as DPO and REBEL across various settings. Furthermore, despite having only 8 billion parameters, Llama-3-8B-it fine-tuned with REFUEL outperforms Llama-3.1-70B-it on long multi-turn dialogues. Implementation of REFUEL can be found at <a class="link-external link-https" href="https://github.com/ZhaolinGao/REFUEL/" rel="external noopener nofollow">this https URL</a>, and models trained by REFUEL can be found at <a class="link-external link-https" href="https://huggingface.co/Cornell-AGI" rel="external noopener nofollow">this https URL</a>.

Efficient Reinforcement Learning Development with RLzoo

AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback

Efficient Reinforcement Learning via Decoupling Exploration and Utilization

SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning

FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning

FinRL-Podracer: High Performance and Scalable Deep Reinforcement Learning for Quantitative Finance

RLIF: Interactive Imitation Learning as Reinforcement Learning

Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration

Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform

Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

A Novel Multi-Step Q-learning Method to Improve Data Efficiency for Deep Reinforcement Learning.

High-throughput Sampling, Communicating and Training for Reinforcement Learning Systems.

KnowRU: Knowledge Reuse Via Knowledge Distillation in Multi-Agent Reinforcement Learning

Parameter Efficient Reinforcement Learning from Human Feedback

FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance

A Framework for Mapping DRL Algorithms with Prioritized Replay Buffer onto Heterogeneous Platforms

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

RL-X: A Deep Reinforcement Learning Library (not only) for RoboCup

KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning