Abstract:The automatic determination of emotional state from multimedia content is an inherently challenging problem with a broad range of applications including biomedical diagnostics, multimedia retrieval, and human computer interfaces. The Audio Video Emotion Challenge (AVEC) 2016 provides a well-defined framework for developing and rigorously evaluating innovative approaches for estimating the arousal and valence states of emotion as a function of time. It presents the opportunity for investigating multimodal solutions that include audio, video, and physiological sensor signals. This paper provides an overview of our AVEC Emotion Challenge system, which uses multi-feature learning and fusion across all available modalities. It includes a number of technical contributions, including the development of novel high- and low-level features for modeling emotion in the audio, video, and physiological channels. Low-level features include modeling arousal in audio with minimal prosodic-based descriptors. High-level features are derived from supervised and unsupervised machine learning approaches based on sparse coding and deep learning. Finally, a state space estimation approach is applied for score fusion that demonstrates the importance of exploiting the time-series nature of the arousal and valence states. The resulting system outperforms the baseline systems [10] on the test evaluation set with an achieved Concordant Correlation Coefficient (CCC) for arousal of 0.770 vs 0.702 (baseline) and for valence of 0.687 vs 0.638. Future work will focus on exploiting the time-varying nature of individual channels in the multi-modal framework.

Exploring Multimodal Visual Features For Continuous Affect Recognition

Emotion Recognition in Videos via Fusing Multimodal Features.

Cross-culture Continuous Emotion Recognition with Multimodal Features

Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition.

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks

Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction

Bagging Based Feature Selection For Dimensional Affect Recognition In The Continuous Emotion Space

Multi-Modal Multi-Cultural Dimensional Continues Emotion Recognition In Dyadic Interactions

Multimodal Affective Dimension Prediction Using Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks.

Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos

Combining Multimodal Features Within A Fusion Network For Emotion Recognition In The Wild

Multimodal Dimensional and Continuous Emotion Recognition in Dyadic Video Interactions.

Multimodal Continuous Emotion Recognition: A Technical Report for ABAW5

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Audio-Video Based Multimodal Emotion Recognition Using Svms and Deep Learning

Valence and Arousal Estimation Based on Multimodal Temporal-Aware Features for Videos in the Wild