Abstract:The rise of intelligent conversation agents, or chatbots, are responsible for the dramatic decrease in remote customer service agent jobs. However, chatbots in their current form, are far from infallible. We theorize that there is an inherent trade-off between a chatbot's response relevance and conversational efficiency in the standard knowledge-bank architecture. Knowledge bank size increases the relevance of successfully queried results, but also increases the difficulty of disambiguating user intents. This inherent trade-off leads to the development of unintelligent fail-safe artifacts such as user confirmations. We argue that, in order to improve user experience and satisfaction, we must decouple knowledge bank size from conversational efficiency. To achieve this, we first design a new artifact that we dub the sequential FAQ (sFAQ) and then evaluate its causal impact on user satisfaction. An sFAQ uses machine learning techniques to first discover common user service journey patterns, then leverage these learned patterns to predict likely subsequent inputs given any focal sequence of inputs. We show that by proactively suggesting potential questions to the user, we can reduce the need for natural language input and thus reduce the need to disambiguate user intent. We then use a novel application of regression discontinuity design (RDD) to study the causal impact of the eliminated reconfirmation dialogues on user satisfaction. Combined, we are able to demonstrate that by eliminating the unintelligent fail-safe artifacts such as user confirmations, the sFAQ will increase satisfaction. Our approach of combining predictive machine learning and causal econometric analysis enables us to open the black box for the underlying causal mechanism linking sFAQ and user satisfaction. This kind of mechanism identification would not be possible even with experimental testing in the field. Our methods and results have useful implications for chatbot applications and user interface design science.

Supervised ensemble sentiment-based framework to measure chatbot quality of services

Enhancing EmoBot: An In-Depth Analysis of User Satisfaction and Faults in an Emotion-Aware Chatbot

Speech Sentiment and Customer Satisfaction Estimation in Socialbot Conversations

Empowering customer satisfaction chatbot using deep learning and sentiment analysis

ASEM: Enhancing Empathy in Chatbot through Attention-based Sentiment and Emotion Modeling

Sentiment Analysis for Personalized Chatbots in E-Commerce Applications

Large-scale Hybrid Approach for Predicting User Satisfaction with Conversational Agents

Investigation of Sentiment Controllable Chatbot

Short Text Conversation Based on Deep Neural Network and Analysis on Evaluation Measures

I, Chatbot: Modeling the Determinants of Users' Satisfaction and Continuance Intention of AI-powered Service Agents.

Emotion and sentiment analysis for intelligent customer service conversation using a multi-task ensemble framework

Scalable Sentiment for Sequence-to-sequence Chatbot Response with Performance Analysis

Evaluating Quality of Chatbots and Intelligent Conversational Agents

A Role-Selected Sharing Network for Joint Machine-Human Chatting Handoff and Service Satisfaction Analysis

Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues

A Data-centric Solution to Improve Online Performance of Customer Service Bots.

Towards highly adaptive Edu-Chatbot

Nowcasting in Chatbot Design: Leveraging Service Journey Patterns to Improve User Satisfaction

CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning

A Two-stage Attention-based Model for Customer Satisfaction Prediction in E-commerce Customer Service

Customer Sentiment Analysis using Weak Supervision for Customer-Agent Chat