Abstract:Intangible cultural heritage (ICH) songs convey folk lives and stories from different communities and nations through touching melodies and lyrics, which are rich in sentiments. Currently, researches about the sentiment analysis of songs are mainly based on lyrics, audios and lyric-audio. Recent studies have shown that deep spectrum features extracted from the spectrogram, generated from the audio, perform well in several speech-based tasks. However, studies combining spectrum features in multimodal sentiment analysis of songs are in a lack. Hence, we propose to combine the audio, lyric and spectrogram to conduct multimodal sentiment analysis for ICH songs, in a tri-modal fusion way. In addition, the correlations and interactions between different modalities are not considered fully. Here, we propose a multimodal song sentiment analysis model (MSSAM), including a strengthened audio features-guided attention (SAFGA) mechanism, which can learn intra- and inter-modal information effectively. First, we obtain strengthened audio features through the fusion of acoustic and spectrum features. Then, the strengthened audio features are used to guide the attention weights distribution of words in the lyric with help of SAFGA, which can make the model focus on the important words with sentiments and related with the sentiment of strengthened audio features, capturing modal interactions and complementary information. We take two world-level ICH lists, Jingju and Kunqu, as examples, and build sentiment analysis datasets. We compare the proposed model with other state-of-the-arts baselines in Jingju and Kunqu datasets. Experimental results demonstrate the superiority of our proposed model.

Lyric-Based Song Sentiment Analysis by Sentiment Vector Space Model

Lyric-based Song Sentiment Classification with Sentiment Vector Space Model.

Sentiment Vector Space Model for Lyric-Based Song Sentiment Classification

Linguistic computing model for music emotion

Sentiment Vector Space Model Based Music Emotion Tag Prediction

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

Joint sentiment analysis of lyrics and audio in music

Reflections on Sentiment/Opinion Analysis

Modelling Emotion Dynamics in Song Lyrics with State Space Models

Improve the application of reinforcement learning and multi‐modal information in music sentiment analysis

Study on Linguistic Computing for Music Emotion

The Contribution of Lyrics and Acoustics to Collaborative Understanding of Mood

Lyric-based Song Emotion Detection with Affective Lexicon and Fuzzy Clustering Method.

A Comparative Study of Teaching Effectiveness in Emotionally Empowered Music Classrooms from a Multimodal Perspective

Text Sentiment Analysis Based on Emotion Adjustment.

A Computational Analysis of Lyric Similarity Perception

Exploring and Applying Audio-Based Sentiment Analysis in Music

Intelligent Analysis of Music's Affective Features and Expressive Pattern

Music's Affective Computing Model Based on Fuzzy Logic

Multimodal Sentiment Analysis of Intangible Cultural Heritage Songs with Strengthened Audio Features-Guided Attention