Abstract:Continuous dimensional emotion recognition from speech helps robots or virtual agents capture the temporal dynamics of a speaker's emotional state in natural human-robot interactions. Temporal modulation cues obtained directly from the time-domain model of auditory perception can better reflect temporal dynamics than the acoustic features usually processed in the frequency domain. Feature extraction, which can reflect temporal dynamics of emotion from temporal modulation cues, is challenging because of the complexity and diversity of the auditory perception model. A recent neuroscientific study suggests that human brains derive multi-resolution representations through temporal modulation analysis. This study investigates multi-resolution representations of an auditory perception model and proposes a novel feature called multi-resolution modulation-filtered cochleagram (MMCG) for predicting valence and arousal values of emotional primitives. The MMCG is constructed by combining four modulation-filtered cochleagrams at different resolutions to capture various temporal and contextual modulation information. In addition, to model the multi-temporal dependencies of the MMCG, we designed a parallel long short-term memory (LSTM) architecture. The results of extensive experiments on the RECOLA and SEWA datasets demonstrate that MMCG provides the best recognition performance in both datasets among all evaluated features. The results also show that the parallel LSTM can build multi-temporal dependencies from the MMCG features, and the performance on valence and arousal prediction is better than that of a plain LSTM method.

Learning Long-Term Temporal Contexts Using Skip RNN for Continuous Emotion Recognition

Efficient Modeling of Long Temporal Contexts for Continuous Emotion Recognition.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Attention-based Temporal Graph Representation Learning for EEG-based Emotion Recognition

Multi-view Domain-Adaptive Representation Learning for EEG-based Emotion Recognition

Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks

Long Short Term Memory Recurrent Neural Network Based Encoding Method for Emotion Recognition in Video.

Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

End-to-End Continuous Emotion Recognition from Video Using 3D Convlstm Networks

Spatial-Temporal Recurrent Neural Network for Emotion Recognition

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Long Short Term Memory Recurrent Neural Network Based Multimodal Dimensional Emotion Recognition

A multimodal fusion-based deep learning framework combined with local-global contextual TCNs for continuous emotion recognition from videos

Advanced LSTM: A Study about Better Time Dependency Modeling in Emotion Recognition

Time-Delay Neural Network for Continuous Emotional Dimension Prediction from Facial Expression Sequences.

Emotion Recognition via Environmental Context and Human Body

Speech Emotion Classification Using Attention-Based LSTM

Spontaneous Speech Emotion Recognition Using Multiscale Deep Convolutional LSTM

An Efficient LSTM Network for Emotion Recognition from Multichannel EEG Signals

Multi-resolution modulation-filtered cochleagram feature for LSTM-based dimensional emotion recognition from speech

Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video