Abstract:Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.

Data Augmentation using Conditional Generative Adversarial Networks for Robust Speech Recognition

Generative Adversarial Networks Based Data Augmentation for Noise Robust Speech Recognition

Data augmentation using generative adversarial networks for robust speech recognition.

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Adversarial Data Augmentation Using VAE-GAN for Disordered Speech Recognition

GANs for Children: A Generative Data Augmentation Strategy for Children Speech Recognition

Conditional Generative Adversarial Networks for Speech Enhancement and Noise-Robust Speaker Verification

Data Augmentation Using Deep Generative Models for Embedding Based Speaker Recognition

Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation

Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition

Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN

Channel-Aware Domain-Adaptive Generative Adversarial Network for Robust Speech Recognition

Improving Speech Emotion Recognition With Adversarial Data Augmentation Network

Study of GANs for Noisy Speech Simulation from Clean Speech

Improving Speech Recognition Using GAN-Based Speech Synthesis and Contrastive Unspoken Text Selection

Double Adversarial Network Based Monaural Speech Enhancement for Robust Speech Recognition.

Generative Adversarial Networks based X-vector Augmentation for Robust Probabilistic Linear Discriminant Analysis in Speaker Verification

Data Augmentation of Room Classifiers using Generative Adversarial Networks

Training Generative Adversarial Network-Based Vocoder with Limited Data Using Augmentation-Conditional Discriminator

Adversarial Data Augmentation for Robust Speaker Verification

Boosting Noise Robustness of Acoustic Model via Deep Adversarial Training