Abstract:The difficulty of emotion recognition in the wild (EmotiW) is how to train a robust model to deal with diverse scenarios and anomalies. The Audio-video Sub-challenge in EmotiW contains audio-video short clips with several emotional labels and the task is to distinguish which label the video belongs to. For the better emotion recognition in videos, we propose a multiple spatio-temporal feature fusion (MSFF) framework, which can more accurately depict emotional information in spatial and temporal dimensions by two mutually complementary sources, including the facial image and audio. The framework is consisted of two parts: the facial image model and the audio model. With respect to the facial image model, three different architectures of spatial-temporal neural networks are employed to extract discriminative features about different emotions in facial expression images. Firstly, the high-level spatial features are obtained by the pre-trained convolutional neural networks (CNN), including VGG-Face and ResNet-50 which are all fed with the images generated by each video. Then, the features of all frames are sequentially input to the Bi-directional Long Short-Term Memory (BLSTM) so as to capture dynamic variations of facial appearance textures in a video. In addition to the structure of CNN-RNN, another spatio-temporal network, namely deep 3-Dimensional Convolutional Neural Networks (3D CNN) by extending the 2D convolution kernel to 3D, is also applied to attain evolving emotional information encoded in multiple adjacent frames. For the audio model, the spectrogram images of speech generated by preprocessing audio, are also modeled in a VGG-BLSTM framework to characterize the affective fluctuation more efficiently. Finally, a fusion strategy with the score matrices of different spatio-temporal networks gained from the above framework is proposed to boost the performance of emotion recognition complementally. Extensive experiments show that the overall accuracy of our proposed MSFF is 60.64%, which achieves a large improvement compared with the baseline and outperform the result of champion team in 2017.

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Emotion Recognition in Videos via Fusing Multimodal Features.

Combining Multimodal Features with Hierarchical Classifier Fusion for Emotion Recognition in the Wild.

Combining Multimodal Features Within A Fusion Network For Emotion Recognition In The Wild

Video Emotion Recognition in the Wild Based on Fusion of Multimodal Features

Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition

Bi-modality Fusion for Emotion Recognition in the Wild.

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

Emotion recognition with multimodal features and temporal models.

Emotion Recognition Using Multimodal Features

Multimodal Emotion Recognition Based on Feature Selection and Extreme Learning Machine in Video Clips.

Multiple Spatio-temporal Feature Learning for Video-based Emotion Recognition in the Wild

Audio And Face Video Emotion Recognition In The Wild Using Deep Neural Networks And Small Datasets

Audio-Video Based Multimodal Emotion Recognition Using Svms and Deep Learning

Multimodal Facial Expression Recognition Based on Dempster-Shafer Theory Fusion Strategy

Multiple Models Fusion for Emotion Recognition in the Wild.

Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

Combining feature-level and decision-level fusion in a hierarchical classifier for emotion recognition in the wild

Multimodal emotion recognition from facial expression and speech based on feature fusion

Audio-visual Emotion Recognition Using Deep Transfer Learning and Multiple Temporal Models.

An autoencoder-based feature level fusion for speech emotion recognition