Abstract:Automatic emotion recognition (AER) systems are burgeoning and systems based on either audio, video, text, or physiological signals have emerged. Multimodal systems, in turn, have shown to improve overall AER accuracy and to also provide some robustness against artifacts and missing data. Collecting multiple signal modalities, however, can be very intrusive, time consuming, and expensive. Recent advances in deep learning based speech-to-text and natural language processing systems, however, have enabled the development of reliable multimodal systems based on speech and text while only requiring the collection of audio data. Audio data, however, is extremely sensitive to environmental disturbances, such as additive noise, thus faces some challenges when deployed “in the wild.” To overcome this issue, speech enhancement algorithms have been deployed at the input signal level to improve testing accuracy in noisy conditions. Speech enhancement algorithms can come in different flavors and can be optimized for different tasks (e.g., for human perception vs. machine performance). Data augmentation, in turn, has also been deployed at the model level during training time to improve accuracy in noisy testing conditions. In this paper, we explore the combination of task-specific speech enhancement and data augmentation as a strategy to improve overall multimodal emotion recognition in noisy conditions. We show that AER accuracy under noisy conditions can be improved to levels close to those seen in clean conditions. When compared against a system without speech enhancement or data augmentation, an increase in AER accuracy of 40% was seen in a cross-corpus test, thus showing promising results for “in the wild” AER.

Automatic Audio Augmentation for Requests Sub-Challenge

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

Adaptive data augmentation for mandarin automatic speech recognition

Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation

A Study on Joint Modeling and Data Augmentation of Multi-Modalities for Audio-Visual Scene Classification

Space-and-speaker-aware Acoustic Modeling with Effective Data Augmentation for Recognition of Multi-Array Conversational Speech

Acoustic Model Ensembling Using Effective Data Augmentation for CHiME-5 Challenge.

Advancing Audio Emotion and Intent Recognition with Large Pre-Trained Models and Bayesian Inference

Auditory-Based Data Augmentation for End-to-End Automatic Speech Recognition

A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection

Autonomous Soundscape Augmentation with Multimodal Fusion of Visual and Participant-linked Inputs

Automated Audio Data Augmentation Network Using Bi-Level Optimization for Sound Event Localization and Detection

Integrating the Data Augmentation Scheme with Various Classifiers for Acoustic Scene Modeling

Acoustic data augmentation for small passive acoustic monitoring datasets

Conferencingspeech Challenge: Towards Far-Field Multi-Channel Speech Enhancement for Video Conferencing

Unsupervised Multi-channel Separation and Adaptation

Task-specific speech enhancement and data augmentation for improved multimodal emotion recognition under noisy conditions

Towards Automatic Data Augmentation for Disordered Speech Recognition

Performance Improvement of Language-Queried Audio Source Separation Based on Caption Augmentation From Large Language Models for DCASE Challenge 2024 Task 9

Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding

Data Augmentation for End-to-end Code-switching Speech Recognition