Abstract:Reliable automatic evaluation of dialogue systems under an interactive environment has long been overdue. An ideal environment for evaluating dialog systems, also known as the Turing test, needs to involve human interaction, which is usually not affordable for large-scale experiments. Though researchers have attempted to use metrics (e.g., perplexity, BLEU) in language generation tasks or some model-based reinforcement learning methods (e.g., self-play evaluation) for automatic evaluation, these methods only show a very weak correlation with the actual human evaluation in practice. To bridge such a gap, we propose a new framework named ENIGMA for estimating human evaluation scores based on recent advances of off-policy evaluation in reinforcement learning. ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation, making automatic evaluations feasible. More importantly, ENIGMA is model-free and agnostic to the behavior policies for collecting the experience data (see details in Section 2), which significantly alleviates the technical difficulties of modeling complex dialogue environments and human behaviors. Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.

Utterance verification based on dynamic garbage evaluation approach

Task Independent Utterance Verification Using Garbage Model Based On Hierarchical Phoneme

An utterance verification algorithm in keyword spotting system

Experimental Investigation into Alignment-based Acoustic Confidence Measures in Keyword Verification for Mandarin Speech

Improving task independent utterance verification based on on-line garbage phoneme likelihood

Confidence Scoring Using Context Dependent Features for Word Verification

A Novel Keyword Verification Algorithm

RESEARCH ON SPEECH KEYWORD VERIFICATION METHOD BASED ON SVM

A Dialogue System Based on Keyword Spotting Technology

Confidence Measures and Integrating Algorithm in Utterance Verification

Utterance Verification Based on Support Vector Machine

Prosodic Features-Based Speaker Verification Using Speaker-Specific-text for Short Utterances.

Fintegrating Confidence Scores for Utterance Verification

A New High-Efficient Language Verification System

Verification of an Incremental Garbage Collector in Hoare-Style Logic

A novel speaker verification approach for certain noisy environment

On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems

Towards Verifiable Text Generation with Evolving Memory and Self-Reflection

Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach

Prosodic Features Based Text-dependent Speaker Recognition with Short Utterance.

Speaker Verification in Agent-Generated Conversations