Abstract:There has been a growing interest in multimodal sentiment analysis and emotion recognition in recent years due to its wide range of practical applications. Multiple modalities allow for the integration of complementary information, improving the accuracy and precision of sentiment and emotion recognition tasks. However, working with multiple modalities presents several challenges, including handling data source heterogeneity, fusing information, aligning and synchronizing modalities, and designing effective feature extraction techniques that capture discriminative information from each modality. This paper introduces a novel framework called "Attention-based Multimodal Sentiment Analysis and Emotion Recognition (AMSAER)" to address these challenges. This framework leverages intra-modality discriminative features and inter-modality correlations in visual, audio, and textual modalities. It incorporates an attention mechanism to facilitate sentiment and emotion classification based on visual, textual, and acoustic inputs by emphasizing relevant aspects of the task. The proposed approach employs separate models for each modality to automatically extract discriminative semantic words, image regions, and audio features. A deep hierarchical model is then developed, incorporating intermediate fusion to learn hierarchical correlations between the modalities at bimodal and trimodal levels. Finally, the framework combines four distinct models through decision-level fusion to enable multimodal sentiment analysis and emotion recognition. The effectiveness of the proposed framework is demonstrated through extensive experiments conducted on the publicly available Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. The results confirm a notable performance improvement compared to state-of-the-art methods, attaining 85% and 93% accuracy for sentiment analysis and emotion classification, respectively. Additionally, when considering class-wise accuracy, the results indicate that the "angry" emotion and "positive" sentiment are classified more effectively than the other emotions and sentiments, achieving 96.80% and 93.14% accuracy, respectively.

Feature-guided Multimodal Sentiment Analysis towards Industry 4.0

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Multimodal Sentiment Analysis: A Survey

Learning Speaker-Independent Multimodal Representation for Sentiment Analysis

Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

Do Photos Help Express Our Feelings: Incorporating Multimodal Features into Microblog Sentiment Analysis.

Integrative Sentiment Analysis: Leveraging Audio, Visual, and Textual Data

Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages

An AutoML-based Approach to Multimodal Image Sentiment Analysis

Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification

MIECF: Multi-faceted information extraction and cross-mixture fusion for multimodal aspect-based sentiment analysis

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

A Multimodal Sentiment Analysis Approach Based on a Joint Chained Interactive Attention Mechanism

Multimodal sentiment analysis based on multiple attention

Multimodal sentimental analysis for social media applications: A comprehensive review

Dual-View Multimodal Interaction in Multimodal Sentiment Analysis

Multimodal Emotion Recognition Based on Feature Fusion.

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

A Survey of Cutting-edge Multimodal Sentiment Analysis