Abstract:Continuous dimensional emotion recognition from speech helps robots or virtual agents capture the temporal dynamics of a speaker's emotional state in natural human-robot interactions. Temporal modulation cues obtained directly from the time-domain model of auditory perception can better reflect temporal dynamics than the acoustic features usually processed in the frequency domain. Feature extraction, which can reflect temporal dynamics of emotion from temporal modulation cues, is challenging because of the complexity and diversity of the auditory perception model. A recent neuroscientific study suggests that human brains derive multi-resolution representations through temporal modulation analysis. This study investigates multi-resolution representations of an auditory perception model and proposes a novel feature called multi-resolution modulation-filtered cochleagram (MMCG) for predicting valence and arousal values of emotional primitives. The MMCG is constructed by combining four modulation-filtered cochleagrams at different resolutions to capture various temporal and contextual modulation information. In addition, to model the multi-temporal dependencies of the MMCG, we designed a parallel long short-term memory (LSTM) architecture. The results of extensive experiments on the RECOLA and SEWA datasets demonstrate that MMCG provides the best recognition performance in both datasets among all evaluated features. The results also show that the parallel LSTM can build multi-temporal dependencies from the MMCG features, and the performance on valence and arousal prediction is better than that of a plain LSTM method.

Multimodal Dimensional Affect Recognition Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks

Multimodal continuous affect recognition based on LSTM and multiple kernel learning

Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks.

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Deep Neural Network And Switching Kalman Filter Based Continuous Affect Recognition

Long Short Term Memory Recurrent Neural Network Based Multimodal Dimensional Emotion Recognition

Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks.

Multimodal Emotion Recognition Using Deep Neural Networks.

Continuous Affect Recognition with Weakly Supervised Learning

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Multimodal Emotional Classification Based on Meaningful Learning

Multimodal Prediction of Affective Dimensions Via Fusing Multiple Regression Techniques

Multimodal Emotion Recognition Using Multimodal Deep Learning

Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition.

A Multimodal Deep Regression Bayesian Network For Affective Video Content Analyses

Multimodal emotion recognition based on deep neural network

A Multi-Task Learning Framework for Emotion Recognition Using 2D Continuous Space.

Emotion Recognition Using Multimodal Residual LSTM Network

Multi-resolution modulation-filtered cochleagram feature for LSTM-based dimensional emotion recognition from speech