Abstract:Audio-visual emotion recognition aims to distinguish human emotional states by integrating the audio and visual data acquired in the expression of emotions. It is crucial for facilitating the affect-related human-machine interaction system by enabling machines to intelligently respond to human emotions. One challenge of this problem is how to efficiently extract feature representations from audio and visual modalities. Although progresses have been made by previous works, most of them ignore common information between audio and visual data during the feature learning process, which may limit the performance since these two modalities are highly correlated in terms of their emotional information. To address this issue, we propose a deep learning approach in order to efficiently utilize common information for audio-visual emotion recognition by correlation analysis. Specifically, we design an audio network and a visual network to extract the feature representations from audio and visual data respectively, and then employ a fusion network to combine the extracted features for emotion prediction. These neural networks are trained by a joint loss, combining: (i) the correlation loss based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation, which extracts common information between audio data, visual data, and the corresponding emotion labels, and (ii) the classification loss, which extracts discriminative information from each modality for emotion prediction. We further generalize our architecture to the semi-supervised learning scenario. The experimental results on the eNTERFACE’05 dataset, BAUM-1s dataset, and RAVDESS dataset show that common information can significantly enhance the stability of features learned from different modalities, and improve the emotion recognition performance.

Emotion-wise feature interaction analysis-based visual emotion distribution learning

Learning Visual Emotion Distributions via Multi-Modal Features Fusion.

Emotional Attention Detection and Correlation Exploration for Image Emotion Distribution Learning.

Facial Emotion Distribution Learning by Exploiting Low-Rank Label Correlations Locally

Capturing Emotion Distribution for Multimedia Emotion Tagging

Approximating Discrete Probability Distribution of Image Emotions by Multi-Modal Features Fusion.

Label Distribution Amendment with Emotional Semantic Correlations for Facial Expression Recognition

Image Emotion Distribution Learning With Graph Convolutional Networks

Multimodal Emotion Distribution Learning.

Discrete Probability Distribution Prediction of Image Emotions with Shared Sparse Learning

A Circular-Structured Representation for Visual Emotion Distribution Learning

Emotion Distribution Learning Based on Peripheral Physiological Signals

Predicting Discrete Probability Distribution of Image Emotions.

Predicting Continuous Probability Distribution Of Image Emotions In Valence-Arousal Space

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Emotion Recognition via Environmental Context and Human Body

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

A Multi-Stage Visual Perception Approach for Image Emotion Analysis

Facial expression distribution prediction based on surface electromyography

StyleEDL: Style-Guided High-order Attention Network for Image Emotion Distribution Learning

Learning Better Representations for Audio-Visual Emotion Recognition with Common Information