Abstract:The automatic determination of emotional state from multimedia content is an inherently challenging problem with a broad range of applications including biomedical diagnostics, multimedia retrieval, and human computer interfaces. The Audio Video Emotion Challenge (AVEC) 2016 provides a well-defined framework for developing and rigorously evaluating innovative approaches for estimating the arousal and valence states of emotion as a function of time. It presents the opportunity for investigating multimodal solutions that include audio, video, and physiological sensor signals. This paper provides an overview of our AVEC Emotion Challenge system, which uses multi-feature learning and fusion across all available modalities. It includes a number of technical contributions, including the development of novel high- and low-level features for modeling emotion in the audio, video, and physiological channels. Low-level features include modeling arousal in audio with minimal prosodic-based descriptors. High-level features are derived from supervised and unsupervised machine learning approaches based on sparse coding and deep learning. Finally, a state space estimation approach is applied for score fusion that demonstrates the importance of exploiting the time-series nature of the arousal and valence states. The resulting system outperforms the baseline systems [10] on the test evaluation set with an achieved Concordant Correlation Coefficient (CCC) for arousal of 0.770 vs 0.702 (baseline) and for valence of 0.687 vs 0.638. Future work will focus on exploiting the time-varying nature of individual channels in the multi-modal framework.

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition.

Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks

Cross-culture Continuous Emotion Recognition with Multimodal Features

Long Short Term Memory Recurrent Neural Network Based Multimodal Dimensional Emotion Recognition

Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks.

Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks.

Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism

Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition

Exploring Multimodal Visual Features For Continuous Affect Recognition

Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction

Multimodal Continuous Prediction of Emotions in Movies using Long Short-Term Memory Networks

Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction

An Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods

Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Multimodal Dimensional and Continuous Emotion Recognition in Dyadic Video Interactions.

Multimodal Prediction of Affective Dimensions Via Fusing Multiple Regression Techniques

Multimodal Transformer Fusion for Continuous Emotion Recognition