Abstract:The rise of intelligent conversation agents, or chatbots, are responsible for the dramatic decrease in remote customer service agent jobs. However, chatbots in their current form, are far from infallible. We theorize that there is an inherent trade-off between a chatbot's response relevance and conversational efficiency in the standard knowledge-bank architecture. Knowledge bank size increases the relevance of successfully queried results, but also increases the difficulty of disambiguating user intents. This inherent trade-off leads to the development of unintelligent fail-safe artifacts such as user confirmations. We argue that, in order to improve user experience and satisfaction, we must decouple knowledge bank size from conversational efficiency. To achieve this, we first design a new artifact that we dub the sequential FAQ (sFAQ) and then evaluate its causal impact on user satisfaction. An sFAQ uses machine learning techniques to first discover common user service journey patterns, then leverage these learned patterns to predict likely subsequent inputs given any focal sequence of inputs. We show that by proactively suggesting potential questions to the user, we can reduce the need for natural language input and thus reduce the need to disambiguate user intent. We then use a novel application of regression discontinuity design (RDD) to study the causal impact of the eliminated reconfirmation dialogues on user satisfaction. Combined, we are able to demonstrate that by eliminating the unintelligent fail-safe artifacts such as user confirmations, the sFAQ will increase satisfaction. Our approach of combining predictive machine learning and causal econometric analysis enables us to open the black box for the underlying causal mechanism linking sFAQ and user satisfaction. This kind of mechanism identification would not be possible even with experimental testing in the field. Our methods and results have useful implications for chatbot applications and user interface design science.

How to Approach Ambiguous Queries in Conversational Search: A Survey of Techniques, Approaches, Tools, and Challenges

A Survey of Conversational Search

Conversational Search with Mixed-Initiative - Asking Good Clarification Questions backed-up by Passage Retrieval

Examining the Potential for Conversational Exploratory Search using a Smart Speaker Digital Assistant

Controlling the Risk of Conversational Search Via Reinforcement Learning

Towards Investigating Biases in Spoken Conversational Search

SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models

Users Meet Clarifying Questions: Toward a Better Understanding of User Interactions for Search Clarification

Query Ambiguity Identification Based On User Behavior Information

PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries

CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search

A Survey on Conversational Recommender Systems

SurveyAgent: A Conversational System for Personalized and Efficient Research Survey

A Survey on Asking Clarification Questions Datasets in Conversational Systems.

Generating Clarifying Questions for Information Retrieval

Using clarification questions to improve software developers' Web search

End-to-End Conversational Search for Online Shopping with Utterance Transfer

Nowcasting in Chatbot Design: Leveraging Service Journey Patterns to Improve User Satisfaction

Towards Conversational Search and Recommendation: System Ask, User Respond.

Simulating and Modeling the Risk of Conversational Search

One Agent Too Many: User Perspectives on Approaches to Multi-agent Conversational AI