Abstract:Conversational search has seen increased recent attention in both the IR andNLP communities. It seeks to clarify and solve users' search needs throughmulti-turn natural language interactions. However, most existing systems aretrained and demonstrated with recorded or artificial conversation logs.Eventually, conversational search systems should be trained, evaluated, anddeployed in an open-ended setting with unseen conversation trajectories. A keychallenge is that training and evaluating such systems both require ahuman-in-the-loop, which is expensive and does not scale. One strategy is tosimulate users, thereby reducing the scaling costs. However, current usersimulators are either limited to only responding to yes-no questions from theconversational search system or unable to produce high-quality responses ingeneral. In this paper, we show that existing user simulation systems could besignificantly improved by a smaller finetuned natural language generationmodel. However, rather than merely reporting it as the new state-of-the-art, weconsider it a strong baseline and present an in-depth investigation ofsimulating user response for conversational search. Our goal is to supplementexisting work with an insightful hand-analysis of unsolved challenges by thebaseline and propose our solutions. The challenges we identified include (1) ablind spot that is difficult to learn, and (2) a specific type of misevaluationin the standard setup. We propose a new generation system to effectively coverthe training blind spot and suggest a new evaluation setup to avoidmisevaluation. Our proposed system leads to significant improvements overexisting systems and large language models such as GPT-4. Additionally, ouranalysis provides insights into the nature of user simulation to facilitatefuture work.

ConvTrans: Transforming Web Search Sessions for Conversational Dense Retrieval.

Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation

CONVERSER: Few-Shot Conversational Dense Retrieval with Synthetic Data Generation

End-to-End Conversational Search for Online Shopping with Utterance Transfer

History-Aware Conversational Dense Retrieval

S2M: Converting Single-Turn to Multi-Turn Datasets for Conversational Question Answering

ConvSDG: Session Data Generation for Conversational Search

Interpreting Conversational Dense Retrieval by Rewriting-Enhanced Inversion of Session Embedding

ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval

TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

Learning Denoised and Interpretable Session Representation for Conversational Search

A Survey of Conversational Search

ConvSearch: A Open-Domain Conversational Search Behavior Dataset

Transfer Meets Hybrid: A Synthetic Approach for Cross-Domain Collaborative Filtering with Text

An In-depth Investigation of User Response Simulation for Conversational Search.

Aligning Query Representation with Rewritten Query and Relevance Judgments in Conversational Search

Curriculum Contrastive Context Denoising for Few-shot Conversational Dense Retrieval

SSP: Self-Supervised Post-training for Conversational Search

Contextualized Query Embeddings for Conversational Search

ConvGQR: Generative Query Reformulation for Conversational Search

Learning to Relate to Previous Turns in Conversational Search