Abstract:Speech encompasses a wealth of information, including but not limited to content, paralinguistic, and environmental information. This comprehensive nature of speech significantly impacts communication and is crucial for human-computer interaction. Chat-Oriented Large Language Models (LLMs), known for their general-purpose assistance capabilities, have evolved to handle multi-modal inputs, including speech. Although these models can be adept at recognizing and analyzing speech, they often fall short of generating appropriate responses. We argue that this is due to the lack of principles on task definition and model development, which requires open-source datasets and metrics suitable for model evaluation. To bridge the gap, we present SD-Eval, a benchmark dataset aimed at multidimensional evaluation of spoken dialogue understanding and generation. SD-Eval focuses on paralinguistic and environmental information and includes 7,303 utterances, amounting to 8.76 hours of speech data. The data is aggregated from eight public datasets, representing four perspectives: emotion, accent, age, and background sound. To assess the SD-Eval benchmark dataset, we implement three different models and construct a training set following a similar process as SD-Eval. The training set contains 1,052.72 hours of speech data and 724.4k utterances. We also conduct a comprehensive evaluation using objective evaluation methods (e.g. BLEU and ROUGE), subjective evaluations and LLM-based metrics for the generated responses. Models conditioned with paralinguistic and environmental information outperform their counterparts in both objective and subjective measures. Moreover, experiments demonstrate LLM-based metrics show a higher correlation with human evaluation compared to traditional metrics. We open-source SD-Eval at <a class="link-external link-https" href="https://github.com/amphionspace/SD-Eval" rel="external noopener nofollow">this https URL</a>.

EMU-SDMS: Advanced speech database management and analysis in R

SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words

An undergraduate Mandarin speech database for speaker recognition research

IMS-Speech: A Speech to Text Tool

A survey of methods to ease the development of highly multilingual text mining applications

Speech Emotion Recognition Based on Formant Characteristics Feature Extraction and Phoneme Type Convergence.

Speech production and perception data collection in R: A tutorial for web-based methods using speechcollectr

A lexical database tool for quantitative phonological research

RescueSpeech: A German Corpus for Speech Recognition in Search and Rescue Domain

DisclosuR: Advancing firm communication analysis through an innovative R package for enhanced textual insights

Studying Mutual Phonetic Influence with a Web-Based Spoken Dialogue System

Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for Polish

Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation

MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages

EmoSpeech: A Corpus of Emotionally Rich and Contextually Detailed Speech Annotations

Emotional Voice Conversion: Theory, Databases and ESD

The latest development of the DELAD project for sharing corpora of speech disorders

Data management to support reproducible research

Praaline: Integrating Tools for Speech Corpus Research

GAUDIE: Development, validation, and exploration of a naturalistic German AUDItory Emotional database

A novel decomposition-based architecture for multilingual speech emotion recognition