Abstract:An ensemble visual-audio emotion recognition framework is proposed based on multi-task and blending learning with multiple features in this paper. To solve the problem that existing features can not accurately identify different emotions, we extract two kinds features, i. e., Interspeech 2010 and deep features for audio data, LBP and deep features for visual data, with the intent to accurately identify different emotions by using different features. Owing to the diversity of features, SVM classifiers and CNN are designed for manual features, i.e., Interspeech 2010 features and local LBP features, and deep features, through which four sub-models are obtained. Finally, the blending ensemble algorithm is used to fuse sub-models to improve the recognition performance of visual-audio emotion recognition. In addition, multi-task learning is applied in the CNN model for deep features, which can predict multiple tasks at the same time with fewer parameters and improve the sensitivity of the single recognition model to user’s emotion by sharing information between different tasks. Experiments are performed using eNTERFACCE database, from which the results indicate that the recognition of multi-task CNN increased by 3% and 2% on average over CNN model in speaker-independent and speaker-dependent experiments, respectively. And emotion recognition accuracy of visual-audio by our method reaches 81.36% and 78.42% in speaker-independent and speaker-dependent experiments, respectively, which maintain higher performance than some state-of-the-art works.

Emotional Tagging Of Videos By Exploring Multiple Emotions' Coexistence

Exploiting Multi-Expression Dependences for Implicit Multi-Emotion Video Tagging.

Multiple Emotional Tagging of Multimedia Data by Exploiting Dependencies among Emotions.

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Implicit Video Multi-Emotion Tagging by Exploiting Multi-Expression Relations.

Hybrid video emotional tagging using users’ EEG and video content

Content-Based Video Emotion Tagging Augmented by Users’ Multiple Physiological Responses

Capturing Emotion Distribution for Multimedia Emotion Tagging

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Implicit Hybrid Video Emotion Tagging By Integrating Video Content And Users' Multiple Physiological Responses

Implicit Video Emotion Tagging from Audiences’ Facial Expression

Multiple Emotion Tagging for Multimedia Data by Exploiting High-Order Dependencies Among Emotions

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Exploiting Multi-Emotion Relations at Feature and Label Levels for Emotion Tagging

Inter-Brain EEG Feature Extraction and Analysis for Continuous Implicit Emotion Tagging During Video Watching

Predicting Emotions in User-Generated Videos.

Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos

Recognizing Emotions Based on Human Actions in Videos.

Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

Mutual Information-Based Emotion Recognition