Abstract:Intangible cultural heritage (ICH) songs convey folk lives and stories from different communities and nations through touching melodies and lyrics, which are rich in sentiments. Currently, researches about the sentiment analysis of songs are mainly based on lyrics, audios and lyric-audio. Recent studies have shown that deep spectrum features extracted from the spectrogram, generated from the audio, perform well in several speech-based tasks. However, studies combining spectrum features in multimodal sentiment analysis of songs are in a lack. Hence, we propose to combine the audio, lyric and spectrogram to conduct multimodal sentiment analysis for ICH songs, in a tri-modal fusion way. In addition, the correlations and interactions between different modalities are not considered fully. Here, we propose a multimodal song sentiment analysis model (MSSAM), including a strengthened audio features-guided attention (SAFGA) mechanism, which can learn intra- and inter-modal information effectively. First, we obtain strengthened audio features through the fusion of acoustic and spectrum features. Then, the strengthened audio features are used to guide the attention weights distribution of words in the lyric with help of SAFGA, which can make the model focus on the important words with sentiments and related with the sentiment of strengthened audio features, capturing modal interactions and complementary information. We take two world-level ICH lists, Jingju and Kunqu, as examples, and build sentiment analysis datasets. We compare the proposed model with other state-of-the-arts baselines in Jingju and Kunqu datasets. Experimental results demonstrate the superiority of our proposed model.

Improve the application of reinforcement learning and multi‐modal information in music sentiment analysis

Automatic Music Emotion Classification Using a New Classification Algorithm

Human-centric Music Medical Therapy Exploration System

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

A New Fuzzy Classifier For Music Emotion Based On Conditional Probability

Real-Time Human-Music Emotional Interaction Based on Deep Learning and Multimodal Sentiment Analysis

Automatic Music Mood Classification by Learning Cross-Media Relevance Between Audio and Lyrics

Real-time Human-Music Emotional Interaction Based on Multimodal Analysis

Music Emotion Research Based on Reinforcement Learning and Multimodal Information

Multimodal Music Mood Classification by Fusion of Audio and Lyrics.

Evaluating Sentiment Similarity Of Songs Based On Social Media Data

Digital Empirical Research of Influencing Factors of Musical Emotion Classification Based on Pleasure-Arousal Musical Emotion Fuzzy Model

Boosting for Multi-Modal Music Emotion Classification.

Enhancing Music Mood Recognition with LLMs and Audio Signal Processing: A Multimodal Approach

Study on Linguistic Computing for Music Emotion

Graph-Based Multimodal Music Mood Classification in Discriminative Latent Space.

Amplifying the music listening experience through song comments on music streaming platforms

Joint sentiment analysis of lyrics and audio in music

Exploring User Emotion in Microblogs for Music Recommendation.

Multimodal Sentiment Analysis of Intangible Cultural Heritage Songs with Strengthened Audio Features-Guided Attention

A Comparative Study of Teaching Effectiveness in Emotionally Empowered Music Classrooms from a Multimodal Perspective