Abstract:Abstract Affective computing is an emerging area of research that aims to enable intelligent systems to recognize, feel, infer and interpret human emotions. The widely spread online and off-line music videos are one of the rich sources of human emotion analysis because it integrates the composer’s internal feeling through song lyrics, musical instruments performance and visual expression. In general, the metadata which music video customers to choose a product includes high-level semantics like emotion so that automatic emotion analysis might be necessary. In this research area, however, the lack of a labeled dataset is a major problem. Therefore, we first construct a balanced music video emotion dataset including diversity of territory, language, culture and musical instruments. We test this dataset over four unimodal and four multimodal convolutional neural networks (CNN) of music and video. First, we separately fine-tuned each pre-trained unimodal CNN and test the performance on unseen data. In addition, we train a 1-dimensional CNN-based music emotion classifier with raw waveform input. The comparative analysis of each unimodal classifier over various optimizers is made to find the best model that can be integrate into a multimodal structure. The best unimodal modality is integrated with corresponding music and video network features for multimodal classifier. The multimodal structure integrates whole music video features and makes final classification with the SoftMax classifier by a late feature fusion strategy. All possible multimodal structures are also combined into one predictive model to get the overall prediction. All the proposed multimodal structure uses cross-validation to overcome the data scarcity problem (overfitting) at the decision level. The evaluation results using various metrics show a boost in the performance of the multimodal architectures compared to each unimodal emotion classifier. The predictive model by integration of all multimodal structure achieves 88.56% in accuracy, 0.88 in f1-score, and 0.987 in area under the curve (AUC) score. The result suggests human high-level emotions are automatically well classified in the proposed CNN-based multimodal networks, even though a small amount of labeled data samples is available for training.

Multiple Music Sentiment Classification Model Based on Convolutional Neural Network.

Automatic Music Emotion Classification Using a New Classification Algorithm

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

A Multimodal Convolutional Neural Network Model for the Analysis of Music Genre on Children's Emotions Influence Intelligence

A Multi-Modal Convolutional Neural Network Model for Intelligent Analysis of the Influence of Music Genres on Children's Emotions

Multimodel Music Emotion Recognition Using Unsupervised Deep Neural Networks

Digital Empirical Research of Influencing Factors of Musical Emotion Classification Based on Pleasure-Arousal Musical Emotion Fuzzy Model

Music emotion recognition based on temporal convolutional attention network using EEG

Multimodal Music Emotion Recognition with Hierarchical Cross-Modal Attention Network

Improve the application of reinforcement learning and multi‐modal information in music sentiment analysis

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

Valence-arousal classification of emotion evoked by Chinese ancient-style music using 1D-CNN-BiLSTM model on EEG signals for college students

Text Sentiment Analysis Based on Convolutional Neural Network and Bidirectional LSTM Model.

A New Fuzzy Classifier For Music Emotion Based On Conditional Probability

Constructing multi-modal emotion recognition model based on convolutional neural network

Constructing a Multimodal Music Teaching Model in College by Integrating Emotions

Music Recommendation System and Recommendation Model Based on Convolutional Neural Network

Combining Vector Space Features and Convolution Neural Network for Text Sentiment Analysis.

Deep learning-based late fusion of multimodal information for emotion classification of music video

Research on Text Sentiment Analysis Based on Neural Network and Ensemble Learning