Abstract:Intangible cultural heritage (ICH) songs convey folk lives and stories from different communities and nations through touching melodies and lyrics, which are rich in sentiments. Currently, researches about the sentiment analysis of songs are mainly based on lyrics, audios and lyric-audio. Recent studies have shown that deep spectrum features extracted from the spectrogram, generated from the audio, perform well in several speech-based tasks. However, studies combining spectrum features in multimodal sentiment analysis of songs are in a lack. Hence, we propose to combine the audio, lyric and spectrogram to conduct multimodal sentiment analysis for ICH songs, in a tri-modal fusion way. In addition, the correlations and interactions between different modalities are not considered fully. Here, we propose a multimodal song sentiment analysis model (MSSAM), including a strengthened audio features-guided attention (SAFGA) mechanism, which can learn intra- and inter-modal information effectively. First, we obtain strengthened audio features through the fusion of acoustic and spectrum features. Then, the strengthened audio features are used to guide the attention weights distribution of words in the lyric with help of SAFGA, which can make the model focus on the important words with sentiments and related with the sentiment of strengthened audio features, capturing modal interactions and complementary information. We take two world-level ICH lists, Jingju and Kunqu, as examples, and build sentiment analysis datasets. We compare the proposed model with other state-of-the-arts baselines in Jingju and Kunqu datasets. Experimental results demonstrate the superiority of our proposed model.

Music sentiment classification integrating audio with lyrics

Multimodal Music Mood Classification by Fusion of Audio and Lyrics.

Automatic Music Emotion Classification Using a New Classification Algorithm

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

Improve the application of reinforcement learning and multi‐modal information in music sentiment analysis

Music Emotion Classification of Chinese Songs based on Lyrics Using TF*IDF and Rhyme.

Boosting for Multi-Modal Music Emotion Classification.

Lyric-based Song Emotion Detection with Affective Lexicon and Fuzzy Clustering Method.

Automatic Music Mood Classification by Learning Cross-Media Relevance Between Audio and Lyrics

Research on Music Emotion Classification Based on Lyrics and Audio

Deep learning based mood tagging for Chinese song lyrics

Graph-Based Multimodal Music Mood Classification in Discriminative Latent Space.

Study on Linguistic Computing for Music Emotion

A Bimodal-based Algorithm for Song Sentiment Classification

Emotional Recognition For Chime Bell Music

Research on Song Sentiment Binary Classification Based on Chinese Lyrics.

Real-Time Human-Music Emotional Interaction Based on Deep Learning and Multimodal Sentiment Analysis

Fine-grained Sentiment Semantic Analysis and Matching of Music and Image

Multimodal Sentiment Analysis of Intangible Cultural Heritage Songs with Strengthened Audio Features-Guided Attention