Abstract:Multimodal sentiment classification is a notable research field that aims to refine sentimental information and classify the sentiment tendency from sequential multimodal data. Most existing sentimental recognition algorithms explore multimodal fusion schemes that achieve good performance. However, there are two key challenges to overcome. First, it is essential to effectively extract inter- and intra-modality features prior to fusion, while simultaneously reducing ambiguity. The second challenge is how to learn modality-invariant representations that capture the underlying similarities. In this paper, we present a modality-invariant temporal learning technique and a new gated inter-modality attention mechanism to overcome these issues. For the first challenge, our proposed gated inter-modality attention mechanism performs modality interactions and filters inconsistencies from multiple modalities in an adaptive manner. We also use parallel structures to learn more comprehensive sentimental information in pairs (i.e., acoustic and visual). In addition, to address the second problem, we treat each modality as a multivariate Gaussian distribution (considering each timestamp as a single Gaussian distribution) and use the KL divergence to capture the implicit temporal distribution-level similarities. These strategies are helpful in reducing domain shifts between different modalities and extracting effective sequential modality-invariant representations. We have conducted experiments on several public datasets (i.e., YouTube and MOUD) and the results show that our proposed method outperforms the state-of-the-art multimodal sentiment categorization methods.

A study of sentiment classification methods with dual model decision fusion

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Multi-feature fusion and dual-channel networks for sentiment analysis

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

An Improved Multimodal Dimension Emotion Recognition Based on Different Fusion Methods

Exploring Multimodal Sentiment Analysis via CBAM Attention and Double-layer BiLSTM Architecture

AFR-BERT: Attention-based mechanism feature relevance fusion multimodal sentiment analysis model

MF-BERT: Multimodal Fusion in Pre-Trained BERT for Sentiment Analysis

Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

Emotion Recognition Model Based on Multimodal Decision Fusion

A Bi-directional Multi-hop Inference Model for Joint Dialog Sentiment Classification and Act Recognition

A multimodal sentiment recognition method based on attention mechanism

Dual Causes Generation Assisted Model for Multimodal Aspect-Based Sentiment Classification

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Multimodal Sentiment Analysis Based on Transformer and Low-rank Fusion

Multimodal Multi-loss Fusion Network for Sentiment Analysis

Multimodal Emotion Recognition Based on Feature Fusion.

Tri-Modalities Fusion for Multimodal Sentiment Analysis

Multi-Feature Fusion Multi-Modal Sentiment Analysis Model Based on Cross-Attention Mechanism