Abstract:Many open-domain dialogue systems rely on multiple response generators, any of which can contribute a response to the dialogue in a particular context. Thus the ability to compare potential responses and then select the best plays an important role in ensuring a dialogue system is coherent and engaging. Dialogue coherence goes beyond simply remaining on topic -- some trivia may be on topic and engaging when mentioned out of the blue, but may not be coherent and grounded in the context of the conversation. We carry out experiments on response selection in the Athena system, an Alexa Prize SocialBot that has dedicated content and multiple topic-specific response generators for a large number of topics. First, we collect a corpus of Athena conversations with live human traffic, where potential responses from all enabled response generators are logged and subsequently annotated for response quality. We compare several off-the-shelf response ranking methods for open-domain dialogue to Athena-Heuristic, a heuristic response ranker that was field-tested in Athena during the third Alexa Prize competition. We also compare these to a transformer-based response ranker we call Athena-RR, that we train on our Athena conversations. Athena-RR uses both the conversational context and the dialogue state to rank the potential responses. We find that Athena-RR with a Recall@1 of 70.79\% outperforms Athena-Heuristic and all of the off-the-shelf rankers by a large margin. We then conduct a live A/B study comparing Athena-Heuristic to Athena-RR in a 6,358 conversations with Alexa users. We show that Athena-RR leads to significantly longer conversations that receive significantly higher user ratings than the heuristic rule-based ranker.

Topic-based Evaluation for Conversational Bots

On Evaluating and Comparing Open Domain Dialog Systems

Short Text Conversation Based on Deep Neural Network and Analysis on Evaluation Measures

BotEval: Facilitating Interactive Human Evaluation

Improving Open-Domain Dialogue Evaluation with a Causal Inference Model

Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents

Dialogue You Can Trust: Human and AI Perspectives on Generated Conversations

On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems

A Transformer-based Response Evaluator for Open-Domain Spoken Conversation

Perspectives for Evaluating Conversational AI

Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation

MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Let's Get Personal: Personal Questions Improve SocialBot Performance in the Alexa Prize

Conversational AI: The Science Behind the Alexa Prize

Psychological Metrics for Dialog System Evaluation

Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems

Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

Toward More Accurate and Generalizable Evaluation Metrics for Task-Oriented Dialogs

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Uncertainty-aware Automatic Evaluation Method for Open-domain Dialogue Systems