Abstract:Multimodal sentiment classification is a notable research field that aims to refine sentimental information and classify the sentiment tendency from sequential multimodal data. Most existing sentimental recognition algorithms explore multimodal fusion schemes that achieve good performance. However, there are two key challenges to overcome. First, it is essential to effectively extract inter- and intra-modality features prior to fusion, while simultaneously reducing ambiguity. The second challenge is how to learn modality-invariant representations that capture the underlying similarities. In this paper, we present a modality-invariant temporal learning technique and a new gated inter-modality attention mechanism to overcome these issues. For the first challenge, our proposed gated inter-modality attention mechanism performs modality interactions and filters inconsistencies from multiple modalities in an adaptive manner. We also use parallel structures to learn more comprehensive sentimental information in pairs (i.e., acoustic and visual). In addition, to address the second problem, we treat each modality as a multivariate Gaussian distribution (considering each timestamp as a single Gaussian distribution) and use the KL divergence to capture the implicit temporal distribution-level similarities. These strategies are helpful in reducing domain shifts between different modalities and extracting effective sequential modality-invariant representations. We have conducted experiments on several public datasets (i.e., YouTube and MOUD) and the results show that our proposed method outperforms the state-of-the-art multimodal sentiment categorization methods.

Graph-Based Multimodal Music Mood Classification in Discriminative Latent Space.

Multimodal Music Mood Classification by Fusion of Audio and Lyrics.

Automatic Music Mood Classification by Learning Cross-Media Relevance Between Audio and Lyrics

Automatic Music Emotion Classification Using a New Classification Algorithm

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Enhancing Music Mood Recognition with LLMs and Audio Signal Processing: A Multimodal Approach

Boosting for Multi-Modal Music Emotion Classification.

Multi-Modal Music Mood Classification Using Co-Training

Improve the application of reinforcement learning and multi‐modal information in music sentiment analysis

Real-time Human-Music Emotional Interaction Based on Multimodal Analysis

Real-Time Human-Music Emotional Interaction Based on Deep Learning and Multimodal Sentiment Analysis

Multimodal Music Emotion Recognition with Hierarchical Cross-Modal Attention Network

Multimodel Music Emotion Recognition Using Unsupervised Deep Neural Networks

Music Mood Classification Based On Lifelog

Mood Classification of Music Using AdaBoost

A Comparative Study of Teaching Effectiveness in Emotionally Empowered Music Classrooms from a Multimodal Perspective

Improving Music Genre Classification from Multi-Modal Properties of Music and Genre Correlations Perspective

Joint sentiment analysis of lyrics and audio in music

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

MMD-MII Model: A Multilayered Analysis and Multimodal Integration Interaction Approach Revolutionizing Music Emotion Classification

A multimodal deep learning algorithm for polyphonic music applied to music sentiment analysis and generation