Abstract:Speech encompasses a wealth of information, including but not limited to content, paralinguistic, and environmental information. This comprehensive nature of speech significantly impacts communication and is crucial for human-computer interaction. Chat-Oriented Large Language Models (LLMs), known for their general-purpose assistance capabilities, have evolved to handle multi-modal inputs, including speech. Although these models can be adept at recognizing and analyzing speech, they often fall short of generating appropriate responses. We argue that this is due to the lack of principles on task definition and model development, which requires open-source datasets and metrics suitable for model evaluation. To bridge the gap, we present SD-Eval, a benchmark dataset aimed at multidimensional evaluation of spoken dialogue understanding and generation. SD-Eval focuses on paralinguistic and environmental information and includes 7,303 utterances, amounting to 8.76 hours of speech data. The data is aggregated from eight public datasets, representing four perspectives: emotion, accent, age, and background sound. To assess the SD-Eval benchmark dataset, we implement three different models and construct a training set following a similar process as SD-Eval. The training set contains 1,052.72 hours of speech data and 724.4k utterances. We also conduct a comprehensive evaluation using objective evaluation methods (e.g. BLEU and ROUGE), subjective evaluations and LLM-based metrics for the generated responses. Models conditioned with paralinguistic and environmental information outperform their counterparts in both objective and subjective measures. Moreover, experiments demonstrate LLM-based metrics show a higher correlation with human evaluation compared to traditional metrics. We open-source SD-Eval at <a class="link-external link-https" href="https://github.com/amphionspace/SD-Eval" rel="external noopener nofollow">this https URL</a>.

An Evaluation of Chinese Human-Computer Dialogue Technology

The First Evaluation of Chinese Human-Computer Dialogue Technology

DialogBench: Evaluating LLMs as Human-like Dialogue Systems

WeChat AI & ICT's Submission for DSTC9 Interactive Dialogue Evaluation Track

HTRDP evaluations on Chinese information processing and intelligent human-machine interface

A Survey of Dialogue System Evaluation.

AliCHI: A Large-scale Multi-modal Dataset and Automated Evaluation Tool for Human-like Dialogue Systems

Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems

Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems

Recent Advances on Human-Computer Dialogue

Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols

Survey of evaluation methods for dialogue systems}{Survey of evaluation methods for dialogue systems

Towards Automatic Evaluation of Customer-Helpdesk Dialogues

SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words

xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark

How to Evaluate Your Dialogue Models: A Review of Approaches

Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

Recent Advances and Challenges in Task-Oriented Dialog Systems

PCDialogEval: Persona and Context Aware Emotional Dialogue Evaluation

Thoughts on Human-Computer Dialogue System

Short Text Conversation Based on Deep Neural Network and Analysis on Evaluation Measures