Abstract:Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. For example, a teacher might try to understand their student's current comprehension level to tailor their instruction accordingly, and a travel agent might ask questions of their customer to understand their preferences in order to recommend activities they might enjoy. LLMs trained with supervised fine-tuning or "single-step" RL, as with standard RLHF, might struggle which tasks that require such goal-directed behavior, since they are not trained to optimize for overall conversational outcomes after multiple turns of interaction. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue. Our key insight is that, though LLMs might not effectively solve goal-directed dialogue tasks out of the box, they can provide useful data for solving such tasks by simulating suboptimal but human-like behaviors. Given a textual description of a goal-directed dialogue task, we leverage LLMs to sample diverse synthetic rollouts of hypothetical in-domain human-human interactions. Our algorithm then utilizes this dataset with offline reinforcement learning to train an interactive conversational agent that can optimize goal-directed objectives over multiple turns. In effect, the LLM produces examples of possible interactions, and RL then processes these examples to learn to perform more optimal interactions. Empirically, we show that our proposed approach achieves state-of-the-art performance in various goal-directed dialogue tasks that include teaching and preference elicitation.

A Zero-Shot Language Agent for Computer Control with Structured Reflection

Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning

AutoAct: Automatic Agent Learning from Scratch for QA Via Self-Planning

Rethinking Task-Oriented Dialogue Systems: From Complex Modularity to Zero-Shot Autonomous Agent

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

Agent3D-Zero: An Agent for Zero-shot 3D Understanding

From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems

Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs

A Zero-Shot approach to the Conversational Tree Search Task

ScreenAgent: A Vision Language Model-driven Computer Control Agent

Devil's Advocate: Anticipatory Reflection for LLM Agents

Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents

Comp-LTL: Temporal Logic Planning via Zero-Shot Policy Composition

Agent Instructs Large Language Models to be General Zero-Shot Reasoners

Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

Language Models as Zero-Shot Trajectory Generators

RL Zero: Zero-Shot Language to Behaviors without any Supervision

AUTOACT: Automatic Agent Learning from Scratch via Self-Planning

Zero-Shot Visual Imitation

Zero-shot Object Navigation with Vision-Language Models Reasoning

Zero-shot Object Navigation with Vision-Language Foundation Models Reasoning.