Abstract:Proactive dialogues serve as a practical yet challenging dialogue problem in the era of large language models (LLMs), where the dialogue policy planning is the key to improving the proactivity of LLMs. Most existing studies enable the dialogue policy planning of LLMs using various prompting schemes or iteratively enhance this capability in handling the given case with verbal AI feedback. However, these approaches are either bounded by the policy planning capability of the frozen LLMs or hard to be transferred to new cases. In this work, we introduce a new dialogue policy planning paradigm to strategize LLMs for proactive dialogue problems with a tunable language model plug-in as a plug-and-play dialogue policy planner, named PPDPP. Specifically, we develop a novel training framework to facilitate supervised fine-tuning over available human-annotated data as well as reinforcement learning from goal-oriented AI feedback with dynamic interaction data collected by the LLM-based self-play simulation. In this manner, the LLM-powered dialogue agent can not only be generalized to different cases after the training, but also be applicable to different applications by just substituting the learned plug-in. In addition, we propose to evaluate the policy planning capability of dialogue systems under the interactive setting. Experimental results demonstrate that PPDPP consistently and substantially outperforms existing approaches on three different proactive dialogue applications, including negotiation, emotional support, and tutoring dialogues.

What problem does this paper attempt to address?

The main problem this paper attempts to address is enhancing the dialogue strategy planning capabilities of large language models (LLMs) in proactive dialogue systems. Specifically, existing methods are either limited by the strategy planning capabilities of frozen LLMs or are difficult to transfer to new scenarios. Additionally, current evaluation methods mainly focus on the quality of single-turn responses and cannot automatically assess strategy planning capabilities in multi-turn dialogues. To tackle these challenges, the authors propose a new dialogue strategy planning paradigm called the "Plug-and-Play Dialogue Policy Planner" (PPDPP). PPDPP improves the strategy planning capabilities of dialogue systems through the following approaches: 1. **Tunable Language Model Plugin**: PPDPP uses a tunable small language model as a plugin responsible for predicting the strategy for the next turn of dialogue. This plugin can be trained through Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to enhance its strategy planning capabilities. 2. **Self-Play Simulation**: PPDPP employs self-play simulation, where two LLMs respectively play the roles of the user and the assistant, simulating a real multi-turn dialogue environment. This approach allows for the collection of dynamic interaction data used to train and optimize the strategy planner. 3. **Goal-Oriented AI Feedback**: PPDPP introduces a third LLM as a reward model to provide goal-oriented natural language feedback, which is then converted into scalar rewards for reinforcement learning. This helps optimize long-term dialogue goals rather than just the quality of single-turn responses. 4. **Interactive Evaluation Method**: To overcome the limitations of traditional evaluation methods, the authors propose an LLM-based interactive evaluation method. This method simulates diverse user-assistant interactions to assess the success rate of the dialogue system and the average number of turns needed to achieve the goals. Experimental results show that PPDPP significantly outperforms existing methods in three different proactive dialogue tasks (including negotiation, emotional support, and coaching dialogues). It not only achieves dialogue goals more efficiently but also flexibly transfers across various application scenarios.

Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

Replicating Complex Dialogue Policy of Humans Via Offline Imitation Learning with Supervised Regularization.

Simulation-Free Hierarchical Latent Policy Planning for Proactive Dialogues

Planning Like Human: A Dual-process Framework for Dialogue Planning

Controllable Conversations: Planning-Based Dialogue Agent with Large Language Models

Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration

Planning with Large Language Models for Conversational Agents

Ask-before-Plan: Proactive Language Agents for Real-World Planning

"Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs

LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy

Hello Again! LLM-powered Personalized Agent for Long-term Dialogue

Script-Based Dialog Policy Planning for LLM-Powered Conversational Agents: A Basic Architecture for an "AI Therapist"

Generating Dialogue Agents via Automated Planning

AdaPlanner: Adaptive Planning from Feedback with Language Models

Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning.

Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization

Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems

Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations

XDAI: A Tuning-free Framework for Exploiting Pre-trained Language Models in Knowledge Grounded Dialogue Generation

Policy Adaptation for Deep Reinforcement Learning-Based Dialogue Management