Abstract:The automatic determination of emotional state from multimedia content is an inherently challenging problem with a broad range of applications including biomedical diagnostics, multimedia retrieval, and human computer interfaces. The Audio Video Emotion Challenge (AVEC) 2016 provides a well-defined framework for developing and rigorously evaluating innovative approaches for estimating the arousal and valence states of emotion as a function of time. It presents the opportunity for investigating multimodal solutions that include audio, video, and physiological sensor signals. This paper provides an overview of our AVEC Emotion Challenge system, which uses multi-feature learning and fusion across all available modalities. It includes a number of technical contributions, including the development of novel high- and low-level features for modeling emotion in the audio, video, and physiological channels. Low-level features include modeling arousal in audio with minimal prosodic-based descriptors. High-level features are derived from supervised and unsupervised machine learning approaches based on sparse coding and deep learning. Finally, a state space estimation approach is applied for score fusion that demonstrates the importance of exploiting the time-series nature of the arousal and valence states. The resulting system outperforms the baseline systems [10] on the test evaluation set with an achieved Concordant Correlation Coefficient (CCC) for arousal of 0.770 vs 0.702 (baseline) and for valence of 0.687 vs 0.638. Future work will focus on exploiting the time-varying nature of individual channels in the multi-modal framework.

Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition.

Multi-Modal Multi-Cultural Dimensional Continues Emotion Recognition In Dyadic Interactions

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Multimodal Dimensional and Continuous Emotion Recognition in Dyadic Video Interactions.

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Exploring Multimodal Visual Features For Continuous Affect Recognition

Cross-culture Continuous Emotion Recognition with Multimodal Features

Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks.

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction

Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks.

Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks

Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition

Multitask Learning and Multistage Fusion for Dimensional Audiovisual Emotion Recognition

Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism

Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video

A Multi-Task Learning Framework for Emotion Recognition Using 2D Continuous Space.

Emotion recognition with multimodal features and temporal models.

Long Short Term Memory Recurrent Neural Network Based Multimodal Dimensional Emotion Recognition

Multi-Task Learning Framework for Emotion Recognition In-the-Wild