Abstract:Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.

Data Augmentation Based on Generative Adversarial Network with Mixed Attention Mechanism

RAGAN: Regression Attention Generative Adversarial Networks

Using GAN to Augment the Synthesizing Images from 3D Models.

Data Augmentation Computing Model Based on Generative Adversarial Network

Pixel Level Data Augmentation for Semantic Image Segmentation Using Generative Adversarial Networks.

Image Augmentations for GAN Training

Emotion Classification with Data Augmentation Using Generative Adversarial Networks

Differentiable Augmentation for Data-Efficient GAN Training

AGANet: Attention-Guided Generative Adversarial Network for Corn Hyperspectral Images Augmentation

AU Data Augmentation Method Based on Generative Adversarial Networks

Diverse Sample Generation with Multi-Branch Conditional Generative Adversarial Network for Remote Sensing Objects Detection.

LatentAugment: Data Augmentation via Guided Manipulation of GAN's Latent Space

Attention-stacked Generative Adversarial Network (AS-GAN)-empowered Sensor Data Augmentation for Online Monitoring of Manufacturing System

Data Augmentation in Emotion Classification Using Generative Adversarial Networks

Data Augmentation Using Conditional GANs for Facial Emotion Recognition

Data Augmentation for Cardiac Magnetic Resonance Image Using Evolutionary GAN

Data Augmentation of Sar Sensor Image via Information Maximizing Generative Adversarial Net

Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN

Data Augmentation Using GANs

MetaCGAN: A Novel GAN Model for Generating High Quality and Diversity Images with Few Training Data

On Data Augmentation for GAN Training