Abstract:Audio-visual emotion recognition aims to distinguish human emotional states by integrating the audio and visual data acquired in the expression of emotions. It is crucial for facilitating the affect-related human-machine interaction system by enabling machines to intelligently respond to human emotions. One challenge of this problem is how to efficiently extract feature representations from audio and visual modalities. Although progresses have been made by previous works, most of them ignore common information between audio and visual data during the feature learning process, which may limit the performance since these two modalities are highly correlated in terms of their emotional information. To address this issue, we propose a deep learning approach in order to efficiently utilize common information for audio-visual emotion recognition by correlation analysis. Specifically, we design an audio network and a visual network to extract the feature representations from audio and visual data respectively, and then employ a fusion network to combine the extracted features for emotion prediction. These neural networks are trained by a joint loss, combining: (i) the correlation loss based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation, which extracts common information between audio data, visual data, and the corresponding emotion labels, and (ii) the classification loss, which extracts discriminative information from each modality for emotion prediction. We further generalize our architecture to the semi-supervised learning scenario. The experimental results on the eNTERFACE’05 dataset, BAUM-1s dataset, and RAVDESS dataset show that common information can significantly enhance the stability of features learned from different modalities, and improve the emotion recognition performance.

The CASIA Audio Emotion Recognition Method for Audio/Visual Emotion Challenge 2011

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Emotion Recognition in Videos via Fusing Multimodal Features.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Audio-Visual Emotion Recognition Based on Facial Expression and Affective Speech

Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation

MEC 2017: Multimodal Emotion Recognition Challenge

Speech Emotion Recognition Based on Linear Discriminant Analysis and Support Vector Machine Decision Tree

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Speech Emotion Recognition Based on Feature Selection and Extreme Learning Machine Decision Tree

Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition

MEC 2016: The Multimodal Emotion Recognition Challenge of CCPR 2016.

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Audio-visual Based Emotion Recognition-a New Approach

Adversarial Domain Adaption for Multi-Cultural Dimensional Emotion Recognition in Dyadic Interactions

An Audiovisual Correlation Matching Method Based on Fine-Grained Emotion and Feature Fusion

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Learning Better Representations for Audio-Visual Emotion Recognition with Common Information