Abstract:Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.

Visual sentiment analysis using data-augmented deep transfer learning techniques

Visual Sentiment Analysis Using Deep Learning Models with Social Media Data

A Deep Features-Based Approach Using Modified ResNet50 and Gradient Boosting for Visual Sentiments Classification

Visual sentiment analysis with semantic correlation enhancement

Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data

Transformer-Based Feature Fusion Approach for Multimodal Visual Sentiment Recognition Using Tweets in the Wild

Leveraging Vision-Language Pre-Trained Model and Contrastive Learning for Enhanced Multimodal Sentiment Analysis

Target-Dependent Multimodal Sentiment Analysis Via Employing Visual-to Emotional-Caption Translation Network using Visual-Caption Pairs

Visual-Textual Sentiment Analysis Enhanced by Hierarchical Cross-Modality Interaction

Deep Visual Semantic Embedding with Text Data Augmentation and Word Embedding Initialization

Visual sentiment analysis based on image caption and adjective–noun–pair description

Sentiment Prediction in Scene Images Via Convolutional Neural Networks

Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN

Holistic Visual-Textual Sentiment Analysis with Prior Models

Semantic-based visual emotion recognition in videos-a transfer learning approach

A deep learning architecture of RA-DLNet for visual sentiment analysis

Novelty fused image and text models based on deep neural network and transformer for multimodal sentiment analysis

From Pixels to Sentiment: Fine-tuning CNNs for Visual Sentiment Prediction

VisdaNet: Visual Distillation and Attention Network for Multimodal Sentiment Classification

Stimuli-Aware Visual Emotion Analysis

A supervised deep learning-based sentiment analysis by the implementation of Word2Vec and GloVe Embedding techniques