Abstract:Conversational search has seen increased recent attention in both the IR andNLP communities. It seeks to clarify and solve users' search needs throughmulti-turn natural language interactions. However, most existing systems aretrained and demonstrated with recorded or artificial conversation logs.Eventually, conversational search systems should be trained, evaluated, anddeployed in an open-ended setting with unseen conversation trajectories. A keychallenge is that training and evaluating such systems both require ahuman-in-the-loop, which is expensive and does not scale. One strategy is tosimulate users, thereby reducing the scaling costs. However, current usersimulators are either limited to only responding to yes-no questions from theconversational search system or unable to produce high-quality responses ingeneral. In this paper, we show that existing user simulation systems could besignificantly improved by a smaller finetuned natural language generationmodel. However, rather than merely reporting it as the new state-of-the-art, weconsider it a strong baseline and present an in-depth investigation ofsimulating user response for conversational search. Our goal is to supplementexisting work with an insightful hand-analysis of unsolved challenges by thebaseline and propose our solutions. The challenges we identified include (1) ablind spot that is difficult to learn, and (2) a specific type of misevaluationin the standard setup. We propose a new generation system to effectively coverthe training blind spot and suggest a new evaluation setup to avoidmisevaluation. Our proposed system leads to significant improvements overexisting systems and large language models such as GPT-4. Additionally, ouranalysis provides insights into the nature of user simulation to facilitatefuture work.

Graph Neural Net-Based User Simulator

A Generative User Simulator with GPT-based Architecture and Goal State Tracking for Reinforced Multi-Domain Dialog Systems

Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems

GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers

GSN: A Graph-Structured Network for Multi-Party Dialogues

An In-depth Investigation of User Response Simulation for Conversational Search.

Jointly Reinforced User Simulator and Task-oriented Dialog System with Simplified Generative Architecture

UserSim: User Simulation via Supervised Generative Adversarial Network

Neural Graph Simulator for Complex Systems

A User Simulator for Task-Completion Dialogues

Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems

Billion-scale Pre-Trained Knowledge Graph Model for Conversational Chatbot

Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems

Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregation

GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation

Towards a Formal Characterization of User Simulation Objectives in Conversational Information Access

Leveraging User Simulation to Develop and Evaluate Conversational Information Access Agents

Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems

Adversarial learning of neural user simulators for dialogue policy optimisation

Training Conversational Agents with Generative Conversational Networks