Abstract:Abstract Affective computing is an emerging area of research that aims to enable intelligent systems to recognize, feel, infer and interpret human emotions. The widely spread online and off-line music videos are one of the rich sources of human emotion analysis because it integrates the composer’s internal feeling through song lyrics, musical instruments performance and visual expression. In general, the metadata which music video customers to choose a product includes high-level semantics like emotion so that automatic emotion analysis might be necessary. In this research area, however, the lack of a labeled dataset is a major problem. Therefore, we first construct a balanced music video emotion dataset including diversity of territory, language, culture and musical instruments. We test this dataset over four unimodal and four multimodal convolutional neural networks (CNN) of music and video. First, we separately fine-tuned each pre-trained unimodal CNN and test the performance on unseen data. In addition, we train a 1-dimensional CNN-based music emotion classifier with raw waveform input. The comparative analysis of each unimodal classifier over various optimizers is made to find the best model that can be integrate into a multimodal structure. The best unimodal modality is integrated with corresponding music and video network features for multimodal classifier. The multimodal structure integrates whole music video features and makes final classification with the SoftMax classifier by a late feature fusion strategy. All possible multimodal structures are also combined into one predictive model to get the overall prediction. All the proposed multimodal structure uses cross-validation to overcome the data scarcity problem (overfitting) at the decision level. The evaluation results using various metrics show a boost in the performance of the multimodal architectures compared to each unimodal emotion classifier. The predictive model by integration of all multimodal structure achieves 88.56% in accuracy, 0.88 in f1-score, and 0.987 in area under the curve (AUC) score. The result suggests human high-level emotions are automatically well classified in the proposed CNN-based multimodal networks, even though a small amount of labeled data samples is available for training.

Deep learning based multimodal emotion recognition using model-level fusion of audio–visual modalities

Emotion Recognition in Videos via Fusing Multimodal Features.

Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning

Multimodal Emotional Classification Based on Meaningful Learning

Multimodal modelling of human emotion using sound, image and text fusion

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Multimodal Emotion Recognition Using Different Fusion Techniques

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

Enhanced multimodal emotion recognition in healthcare analytics: A deep learning based model-level fusion approach

Multitask Learning and Multistage Fusion for Dimensional Audiovisual Emotion Recognition

Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video

Speech emotion recognition using multimodal feature fusion with machine learning approach

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

Multimodal Emotion Recognition Framework Using a Decision-Level Fusion and Feature-Level Fusion Approach

Audio-Visual Fusion Network Based on Conformer for Multimodal Emotion Recognition

Multimodal Deep Convolutional Neural Network for Audio-Visual Emotion Recognition.

Deep learning-based late fusion of multimodal information for emotion classification of music video

Multimodal Speech Emotion Recognition Using Modality-specific Self-Supervised Frameworks

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals