Abstract:Today, most large-scale conversational AI agents (e.g. Alexa, Siri, or Google Assistant) are built using manually annotated data to train the different components of the system. Typically, the accuracy of the ML models in these components are improved by manually transcribing and annotating data. As the scope of these systems increase to cover more scenarios and domains, manual annotation to improve the accuracy of these components becomes prohibitively costly and time consuming. In this paper, we propose a system that leverages user-system interaction feedback signals to automate learning without any manual annotation. Users here tend to modify a previous query in hopes of fixing an error in the previous turn to get the right results. These reformulations, which are often preceded by defective experiences caused by errors in ASR, NLU, ER or the application. In some cases, users may not properly formulate their requests (e.g. providing partial title of a song), but gleaning across a wider pool of users and sessions reveals the underlying recurrent patterns. Our proposed self-learning system automatically detects the errors, generate reformulations and deploys fixes to the runtime system to correct different types of errors occurring in different components of the system. In particular, we propose leveraging an absorbing Markov Chain model as a collaborative filtering mechanism in a novel attempt to mine these patterns. We show that our approach is highly scalable, and able to learn reformulations that reduce Alexa-user errors by pooling anonymized data across millions of customers. The proposed self-learning system achieves a win/loss ratio of 11.8 and effectively reduces the defect rate by more than 30% on utterance level reformulations in our production A/B tests. To the best of our knowledge, this is the first self-learning large-scale conversational AI system in production.

System-Level Natural Language Feedback

Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

Let Me Teach You: Pedagogical Foundations of Feedback for Language Models

Iterative Design and Classroom Evaluation of Automated Formative Feedback for Improving Peer Feedback Localization

Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing

Learning from Naturally Occurring Feedback

UltraFeedback: Boosting Language Models with Scaled AI Feedback

WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

Aligning Large Language Models from Self-Reference AI Feedback with one General Principle

Feedback-Based Self-Learning in Large-Scale Conversational AI Agents

Using Large Language Models to Provide Explanatory Feedback to Human Tutors

The Future of Open Human Feedback

SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text

Large Language Models Are Active Critics in NLG Evaluation

On the Automated Processing of User Feedback

Large Language Models Meet User Interfaces: The Case of Provisioning Feedback

Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models

Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

Improving Code Generation by Training with Natural Language Feedback