Abstract:The difficulty of emotion recognition in the wild (EmotiW) is how to train a robust model to deal with diverse scenarios and anomalies. The Audio-video Sub-challenge in EmotiW contains audio-video short clips with several emotional labels and the task is to distinguish which label the video belongs to. For the better emotion recognition in videos, we propose a multiple spatio-temporal feature fusion (MSFF) framework, which can more accurately depict emotional information in spatial and temporal dimensions by two mutually complementary sources, including the facial image and audio. The framework is consisted of two parts: the facial image model and the audio model. With respect to the facial image model, three different architectures of spatial-temporal neural networks are employed to extract discriminative features about different emotions in facial expression images. Firstly, the high-level spatial features are obtained by the pre-trained convolutional neural networks (CNN), including VGG-Face and ResNet-50 which are all fed with the images generated by each video. Then, the features of all frames are sequentially input to the Bi-directional Long Short-Term Memory (BLSTM) so as to capture dynamic variations of facial appearance textures in a video. In addition to the structure of CNN-RNN, another spatio-temporal network, namely deep 3-Dimensional Convolutional Neural Networks (3D CNN) by extending the 2D convolution kernel to 3D, is also applied to attain evolving emotional information encoded in multiple adjacent frames. For the audio model, the spectrogram images of speech generated by preprocessing audio, are also modeled in a VGG-BLSTM framework to characterize the affective fluctuation more efficiently. Finally, a fusion strategy with the score matrices of different spatio-temporal networks gained from the above framework is proposed to boost the performance of emotion recognition complementally. Extensive experiments show that the overall accuracy of our proposed MSFF is 60.64%, which achieves a large improvement compared with the baseline and outperform the result of champion team in 2017.

DBLSTM-based Multi-Scale Fusion for Dynamic Emotion Prediction in Music.

A Deep Bidirectional Long Short-Term Memory Based Multi-Scale Approach for Music Dynamic Emotion Prediction

Svr Based Double-Scale Regression For Dynamic Emotion Prediction In Music

Multi-scale Context Based Attention for Dynamic Music Emotion Prediction

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Multi-Scale Approaches to the MediaEval 2015 "emotion in Music" Task.

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

ADFF: Attention Based Deep Feature Fusion Approach for Music Emotion Recognition

A Multimodal Framework for Large-Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

BLNN:a muscular and tall architecture for emotion prediction in music

Spontaneous Speech Emotion Recognition Using Multiscale Deep Convolutional LSTM

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

Multi-scale spatiotemporal representation learning for EEG-based emotion recognition

Music emotion recognition based on temporal convolutional attention network using EEG

Multiple Spatio-temporal Feature Learning for Video-based Emotion Recognition in the Wild