Abstract:There has been a growing interest in multimodal sentiment analysis and emotion recognition in recent years due to its wide range of practical applications. Multiple modalities allow for the integration of complementary information, improving the accuracy and precision of sentiment and emotion recognition tasks. However, working with multiple modalities presents several challenges, including handling data source heterogeneity, fusing information, aligning and synchronizing modalities, and designing effective feature extraction techniques that capture discriminative information from each modality. This paper introduces a novel framework called "Attention-based Multimodal Sentiment Analysis and Emotion Recognition (AMSAER)" to address these challenges. This framework leverages intra-modality discriminative features and inter-modality correlations in visual, audio, and textual modalities. It incorporates an attention mechanism to facilitate sentiment and emotion classification based on visual, textual, and acoustic inputs by emphasizing relevant aspects of the task. The proposed approach employs separate models for each modality to automatically extract discriminative semantic words, image regions, and audio features. A deep hierarchical model is then developed, incorporating intermediate fusion to learn hierarchical correlations between the modalities at bimodal and trimodal levels. Finally, the framework combines four distinct models through decision-level fusion to enable multimodal sentiment analysis and emotion recognition. The effectiveness of the proposed framework is demonstrated through extensive experiments conducted on the publicly available Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. The results confirm a notable performance improvement compared to state-of-the-art methods, attaining 85% and 93% accuracy for sentiment analysis and emotion classification, respectively. Additionally, when considering class-wise accuracy, the results indicate that the "angry" emotion and "positive" sentiment are classified more effectively than the other emotions and sentiments, achieving 96.80% and 93.14% accuracy, respectively.

Emotion detection using convolutional neural network and long short-term memory: a deep multimodal framework

Non-Acted Text and Keystrokes Database and Learning Methods to Recognize Emotions

A Deep Neural Network Model for the Detection and Classification of Emotions from Textual Content

Novel multimodal emotion detection method using Electroencephalogram and Electrocardiogram signals

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Enhancing Emotion Detection with Non-invasive Multi-Channel EEG and Hybrid Deep Learning Architecture

Text-Based Emotion Recognition Using Deep Learning Approach

M1M2: Deep-Learning-Based Real-Time Emotion Recognition from Neural Activity

Multimodal Emotion Recognition Using Deep Neural Networks.

Exploring Emotions in EEG: Deep Learning Approach with Feature Fusion

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Design of Hybrid Deep Learning Model for Text-based Emotion Recognition

Deploying Machine Learning Techniques for Human Emotion Detection

Deep Learning-Based Approach for Emotion Recognition Using Electroencephalography (EEG) Signals Using Bi-Directional Long Short-Term Memory (Bi-LSTM)

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

S-LSTM-ATT: a hybrid deep learning approach with optimized features for emotion recognition in electroencephalogram

Human–Computer Interaction with a Real-Time Speech Emotion Recognition with Ensembling Techniques 1D Convolution Neural Network and Attention

Emotion Recognition From Speech and Text using Long Short-Term Memory

Smart Multi-Model Emotion Recognition System with Deep learning

A real time face emotion classification and recognition using deep learning model