Abstract:There has been a growing interest in multimodal sentiment analysis and emotion recognition in recent years due to its wide range of practical applications. Multiple modalities allow for the integration of complementary information, improving the accuracy and precision of sentiment and emotion recognition tasks. However, working with multiple modalities presents several challenges, including handling data source heterogeneity, fusing information, aligning and synchronizing modalities, and designing effective feature extraction techniques that capture discriminative information from each modality. This paper introduces a novel framework called "Attention-based Multimodal Sentiment Analysis and Emotion Recognition (AMSAER)" to address these challenges. This framework leverages intra-modality discriminative features and inter-modality correlations in visual, audio, and textual modalities. It incorporates an attention mechanism to facilitate sentiment and emotion classification based on visual, textual, and acoustic inputs by emphasizing relevant aspects of the task. The proposed approach employs separate models for each modality to automatically extract discriminative semantic words, image regions, and audio features. A deep hierarchical model is then developed, incorporating intermediate fusion to learn hierarchical correlations between the modalities at bimodal and trimodal levels. Finally, the framework combines four distinct models through decision-level fusion to enable multimodal sentiment analysis and emotion recognition. The effectiveness of the proposed framework is demonstrated through extensive experiments conducted on the publicly available Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. The results confirm a notable performance improvement compared to state-of-the-art methods, attaining 85% and 93% accuracy for sentiment analysis and emotion classification, respectively. Additionally, when considering class-wise accuracy, the results indicate that the "angry" emotion and "positive" sentiment are classified more effectively than the other emotions and sentiments, achieving 96.80% and 93.14% accuracy, respectively.

MeToo Tweets Sentiment Analysis Using Multi Modal frameworks

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

#MeTooMA: Multi-Aspect Annotations of Tweets Related to the MeToo Movement

A Multitask Multimodal Ensemble Model for Sentiment- and Emotion-Aided Tweet Act Classification

A soft voting ensemble learning-based approach for multimodal sentiment analysis

A Multimodal Framework for the Detection of Hateful Memes

Multimodal Sentiment Analysis Based on Recurrent Neural Network and Multimodal Attention.

Multimodal sentiment analysis leveraging the strength of deep neural networks enhanced by the XGBoost classifier

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning

M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

Multi-Model Fusion Framework Using Deep Learning for Visual-Textual Sentiment Classification

Team Neuro at SemEval-2020 Task 8: Multi-Modal Fine Grain Emotion Classification of Memes using Multitask Learning

Two-Way Feature Extraction Using Sequential and Multimodal Approach for Hateful Meme Classification

Multimodal Sentiment Analysis: Perceived vs Induced Sentiments

Harnessing Twitter for Automatic Sentiment Identification Using Machine Learning Techniques

MBi-GRUMCONV: A novel Multi Bi-GRU and Multi CNN-Based deep learning model for social media sentiment analysis

Gated Mechanism for Attention Based Multi Modal Sentiment Analysis

MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis To Explore the Structure of Emotions

Me Too Movement Sentiment Analysis