Abstract:The rise of intelligent conversation agents, or chatbots, are responsible for the dramatic decrease in remote customer service agent jobs. However, chatbots in their current form, are far from infallible. We theorize that there is an inherent trade-off between a chatbot's response relevance and conversational efficiency in the standard knowledge-bank architecture. Knowledge bank size increases the relevance of successfully queried results, but also increases the difficulty of disambiguating user intents. This inherent trade-off leads to the development of unintelligent fail-safe artifacts such as user confirmations. We argue that, in order to improve user experience and satisfaction, we must decouple knowledge bank size from conversational efficiency. To achieve this, we first design a new artifact that we dub the sequential FAQ (sFAQ) and then evaluate its causal impact on user satisfaction. An sFAQ uses machine learning techniques to first discover common user service journey patterns, then leverage these learned patterns to predict likely subsequent inputs given any focal sequence of inputs. We show that by proactively suggesting potential questions to the user, we can reduce the need for natural language input and thus reduce the need to disambiguate user intent. We then use a novel application of regression discontinuity design (RDD) to study the causal impact of the eliminated reconfirmation dialogues on user satisfaction. Combined, we are able to demonstrate that by eliminating the unintelligent fail-safe artifacts such as user confirmations, the sFAQ will increase satisfaction. Our approach of combining predictive machine learning and causal econometric analysis enables us to open the black box for the underlying causal mechanism linking sFAQ and user satisfaction. This kind of mechanism identification would not be possible even with experimental testing in the field. Our methods and results have useful implications for chatbot applications and user interface design science.

Developing a Llama-Based Chatbot for CI/CD Question Answering: A Case Study at Ericsson

Uman-in-thel oop

Teaching Machines to Converse

Expanding Chatbot Knowledge in Customer Service: Context-Aware Similar Question Generation Using Large Language Models

Towards Designing a Question-Answering Chatbot for Online News: Understanding Questions and Perspectives

Learnings from a Large-Scale Deployment of an LLM-Powered Expert-in-the-Loop Healthcare Chatbot

Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop

RAG based Question-Answering for Contextual Response Prediction System

Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering

Beyond-RAG: Question Identification and Answer Generation in Real-Time Conversations

Organization-Wide Continuous Learning (OWCL): Personalized AI Chatbots for Effective Post-Training Knowledge Retention

From Questions to Insightful Answers: Building an Informed Chatbot for University Resources

Nowcasting in Chatbot Design: Leveraging Service Journey Patterns to Improve User Satisfaction

"Always Nice and Confident, Sometimes wrong": Developer's Experiences Engaging Generative AI Chatbots Versus Human-Powered Q&A Platforms

Hi, How Can I Help You?: Automating Enterprise IT Support Help Desks

Reinforcement Learning for Optimizing RAG for Domain Chatbots

Walert: Putting Conversational Search Knowledge into Action by Building and Evaluating a Large Language Model-Powered Chatbot

Deep Learning for Intelligent Customer Service Automation: Development of GRU, LSTM, and Recurrent Neural Network Architectures for Chatbot Applications

Deep Learning Based Amharic Chatbot for FAQs in Universities

Enhancing Supermarket Robot Interaction: A Multi-Level LLM Conversational Interface for Handling Diverse Customer Intents

Ask-EDA: A Design Assistant Empowered by LLM, Hybrid RAG and Abbreviation De-hallucination