Abstract:Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. For example, a teacher might try to understand their student's current comprehension level to tailor their instruction accordingly, and a travel agent might ask questions of their customer to understand their preferences in order to recommend activities they might enjoy. LLMs trained with supervised fine-tuning or "single-step" RL, as with standard RLHF, might struggle which tasks that require such goal-directed behavior, since they are not trained to optimize for overall conversational outcomes after multiple turns of interaction. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue. Our key insight is that, though LLMs might not effectively solve goal-directed dialogue tasks out of the box, they can provide useful data for solving such tasks by simulating suboptimal but human-like behaviors. Given a textual description of a goal-directed dialogue task, we leverage LLMs to sample diverse synthetic rollouts of hypothetical in-domain human-human interactions. Our algorithm then utilizes this dataset with offline reinforcement learning to train an interactive conversational agent that can optimize goal-directed objectives over multiple turns. In effect, the LLM produces examples of possible interactions, and RL then processes these examples to learn to perform more optimal interactions. Empirically, we show that our proposed approach achieves state-of-the-art performance in various goal-directed dialogue tasks that include teaching and preference elicitation.

LAD: Language Models as Data for Zero-Shot Dialog

LaDA: Latent Dialogue Action For Zero-shot Cross-lingual Neural Network Language Modeling

Zero-shot Slot Filling in the Age of LLMs for Dialogue Systems

LUCID: LLM-Generated Utterances for Complex and Interesting Dialogues

Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems

LaMDA: Language Models for Dialog Applications

Training Zero-Shot Generalizable End-to-End Task-Oriented Dialog System Without Turn-level Dialog Annotations

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

Towards a Zero-Data, Controllable, Adaptive Dialog System

Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Tracking

LLM-powered Zero-shot Online Log Parsing

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

An Approach to Build Zero-Shot Slot-Filling System for Industry-Grade Conversational Assistants

Few-Shot Dialogue Generation Without Annotated Data: A Transfer Learning Approach

Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search

StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data

Synthetic Dialogue Dataset Generation using LLM Agents

ChatZero:Zero-shot Cross-Lingual Dialogue Generation via Pseudo-Target Language

Towards Zero-Label Language Learning

Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selection