An In-depth Investigation of User Response Simulation for Conversational Search.

Zhenduo Wang,Zhichao Xu,Vivek Srikumar,Qingyao Ai

DOI: https://doi.org/10.1145/3589334.3645447

2024-01-01

Abstract:Conversational search has seen increased recent attention in both the IR andNLP communities. It seeks to clarify and solve users' search needs throughmulti-turn natural language interactions. However, most existing systems aretrained and demonstrated with recorded or artificial conversation logs.Eventually, conversational search systems should be trained, evaluated, anddeployed in an open-ended setting with unseen conversation trajectories. A keychallenge is that training and evaluating such systems both require ahuman-in-the-loop, which is expensive and does not scale. One strategy is tosimulate users, thereby reducing the scaling costs. However, current usersimulators are either limited to only responding to yes-no questions from theconversational search system or unable to produce high-quality responses ingeneral. In this paper, we show that existing user simulation systems could besignificantly improved by a smaller finetuned natural language generationmodel. However, rather than merely reporting it as the new state-of-the-art, weconsider it a strong baseline and present an in-depth investigation ofsimulating user response for conversational search. Our goal is to supplementexisting work with an insightful hand-analysis of unsolved challenges by thebaseline and propose our solutions. The challenges we identified include (1) ablind spot that is difficult to learn, and (2) a specific type of misevaluationin the standard setup. We propose a new generation system to effectively coverthe training blind spot and suggest a new evaluation setup to avoidmisevaluation. Our proposed system leads to significant improvements overexisting systems and large language models such as GPT-4. Additionally, ouranalysis provides insights into the nature of user simulation to facilitatefuture work.