EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark

Ziyang Ma,Mingjie Chen,Hezhao Zhang,Zhisheng Zheng,Wenxi Chen,Xiquan Li,Jiaxin Ye,Xie Chen,Thomas Hain

2024-06-11

Abstract:Speech emotion recognition (SER) is an important part of human-computer interaction, receiving extensive attention from both industry and academia. However, the current research field of SER has long suffered from the following problems: 1) There are few reasonable and universal splits of the datasets, making comparing different models and methods difficult. 2) No commonly used benchmark covers numerous corpus and languages for researchers to refer to, making reproduction a burden. In this paper, we propose EmoBox, an out-of-the-box multilingual multi-corpus speech emotion recognition toolkit, along with a benchmark for both intra-corpus and cross-corpus settings. For intra-corpus settings, we carefully designed the data partitioning for different datasets. For cross-corpus settings, we employ a foundation SER model, emotion2vec, to mitigate annotation errors and obtain a test set that is fully balanced in speakers and emotions distributions. Based on EmoBox, we present the intra-corpus SER results of 10 pre-trained speech models on 32 emotion datasets with 14 languages, and the cross-corpus SER results on 4 datasets with the fully balanced test sets. To the best of our knowledge, this is the largest SER benchmark, across language scopes and quantity scales. We hope that our toolkit and benchmark can facilitate the research of SER in the community.

Sound,Artificial Intelligence,Computation and Language,Multimedia,Audio and Speech Processing

What problem does this paper attempt to address?

The paper aims to address two main issues in the field of Speech Emotion Recognition (SER):

1. **Unreasonable and Inconsistent Dataset Division**: The current SER research field lacks rational and universal methods for dataset division, making it difficult to compare different models and methods.

2. **Lack of Comprehensive Multilingual and Multicorpus Benchmarks**: There is currently no widely used benchmark that covers multiple corpora and languages, which burdens researchers with repetitive work and limits the reproducibility of research results.

To solve these problems, the authors propose EmoBox—a multilingual, multicorpus speech emotion recognition toolkit and benchmark. The main contributions of EmoBox include:

- **Toolkit**: Provides an easy-to-use toolkit that supports multilingual and multicorpus speech emotion recognition research.

- **Data Preparation and Division**: Systematically classifies and divides 32 emotional datasets, ensuring effective utilization of the datasets and designing reasonable training/testing set division schemes for different datasets.

- **Benchmark Establishment**: Establishes the largest-scale speech emotion recognition benchmark to date, covering 14 different languages in 32 datasets, as well as comprehensive balanced test sets under 4 cross-corpus settings.

Through these efforts, EmoBox aims to provide a powerful toolkit and benchmark for the field of speech emotion recognition, promoting research progress, enhancing model comparability, and ultimately driving technological innovation in the field.