Abstract:Abstract Affective computing is an emerging area of research that aims to enable intelligent systems to recognize, feel, infer and interpret human emotions. The widely spread online and off-line music videos are one of the rich sources of human emotion analysis because it integrates the composer’s internal feeling through song lyrics, musical instruments performance and visual expression. In general, the metadata which music video customers to choose a product includes high-level semantics like emotion so that automatic emotion analysis might be necessary. In this research area, however, the lack of a labeled dataset is a major problem. Therefore, we first construct a balanced music video emotion dataset including diversity of territory, language, culture and musical instruments. We test this dataset over four unimodal and four multimodal convolutional neural networks (CNN) of music and video. First, we separately fine-tuned each pre-trained unimodal CNN and test the performance on unseen data. In addition, we train a 1-dimensional CNN-based music emotion classifier with raw waveform input. The comparative analysis of each unimodal classifier over various optimizers is made to find the best model that can be integrate into a multimodal structure. The best unimodal modality is integrated with corresponding music and video network features for multimodal classifier. The multimodal structure integrates whole music video features and makes final classification with the SoftMax classifier by a late feature fusion strategy. All possible multimodal structures are also combined into one predictive model to get the overall prediction. All the proposed multimodal structure uses cross-validation to overcome the data scarcity problem (overfitting) at the decision level. The evaluation results using various metrics show a boost in the performance of the multimodal architectures compared to each unimodal emotion classifier. The predictive model by integration of all multimodal structure achieves 88.56% in accuracy, 0.88 in f1-score, and 0.987 in area under the curve (AUC) score. The result suggests human high-level emotions are automatically well classified in the proposed CNN-based multimodal networks, even though a small amount of labeled data samples is available for training.

Mood Detection in Aesthetically Appealing Video Based on Color Association

Image Aesthetics Assessment With Emotion-Aware Multibranch Network

Classification of Aesthetic Natural Scene Images Using Statistical and Semantic Features

Optimized Ensemble Machine Learning Approach for Emotion Detection from Thermal Images

A VGG16 Based Hybrid Deep Convolutional Neural Network Based Real-Time Video Frame Emotion Detection System for Affective Human Computer Interaction

Recognition of Emotion According to the Physical Elements of the Video

Beauty is Here

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Video Classification and Recommendation Based on Affective Analysis of Viewers

Students’ affective content analysis in smart classroom environment using deep learning techniques

Deep learning-based late fusion of multimodal information for emotion classification of music video

Deep Residual Adaptive Neural Network Based Feature Extraction for Cognitive Computing with Multimodal Sentiment Sensing and Emotion Recognition Process

Decoding viewer emotions in video ads

Human Color Perception: The Impact of Color Perception on Fine-Grained Emotion Prediction in Movie and Television Videos

Beauty is here: evaluating aesthetics in videos using multimodal features and free training data.

Deep Convolution Network Based Emotion Analysis for Automatic Detection of Mild Cognitive Impairment in the Elderly

Automatic Detection of Sentimentality from Facial Expressions

Micro-Facial Expression Recognition in Video Based on Optimal Convolutional Neural Network (MFEOCNN) Algorithm

An enhanced fitness function to recognize unbalanced human emotions data

Emotion recognition techniques with rule based and machine learning approaches

An Ensemble Approach for Facial Expression Analysis in Video