Abstract:There has been a growing interest in multimodal sentiment analysis and emotion recognition in recent years due to its wide range of practical applications. Multiple modalities allow for the integration of complementary information, improving the accuracy and precision of sentiment and emotion recognition tasks. However, working with multiple modalities presents several challenges, including handling data source heterogeneity, fusing information, aligning and synchronizing modalities, and designing effective feature extraction techniques that capture discriminative information from each modality. This paper introduces a novel framework called "Attention-based Multimodal Sentiment Analysis and Emotion Recognition (AMSAER)" to address these challenges. This framework leverages intra-modality discriminative features and inter-modality correlations in visual, audio, and textual modalities. It incorporates an attention mechanism to facilitate sentiment and emotion classification based on visual, textual, and acoustic inputs by emphasizing relevant aspects of the task. The proposed approach employs separate models for each modality to automatically extract discriminative semantic words, image regions, and audio features. A deep hierarchical model is then developed, incorporating intermediate fusion to learn hierarchical correlations between the modalities at bimodal and trimodal levels. Finally, the framework combines four distinct models through decision-level fusion to enable multimodal sentiment analysis and emotion recognition. The effectiveness of the proposed framework is demonstrated through extensive experiments conducted on the publicly available Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. The results confirm a notable performance improvement compared to state-of-the-art methods, attaining 85% and 93% accuracy for sentiment analysis and emotion classification, respectively. Additionally, when considering class-wise accuracy, the results indicate that the "angry" emotion and "positive" sentiment are classified more effectively than the other emotions and sentiments, achieving 96.80% and 93.14% accuracy, respectively.

AMINN - Attention-Based Multi-Information Neural Network for Emotion Recognition.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

AHRNN: Attention‐Based Hybrid Robust Neural Network for Emotion Recognition

An improved multi-input deep convolutional neural network for automatic emotion recognition

Attention-Based BiLSTM Network with Lexical Feature for Emotion Classification

Deep Emotional Arousal Network for Multimodal Sentiment Analysis and Emotion Recognition

Multi-head attention fusion networks for multi-modal speech emotion recognition

A multi-modal and multi-scale emotion-enhanced inference model based on fuzzy recognition

Multiview Feature Fusion Attention Convolutional Recurrent Neural Networks for EEG-Based Emotion Recognition

Multi-modal fusion network with complementarity and importance for emotion recognition

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Concept-guided multi-level attention network for image emotion recognition

Multiscale Emotion Representation Learning for Affective Image Recognition

Multimodal Physiological Signal Emotion Recognition Based on Convolutional Recurrent Neural Network

Facial Emotion Recognition Combining Auxiliary Classifiers and Multiscale CBAM Attention Mechanisms

A novel signal channel attention network for multi-modal emotion recognition

A Model for EEG-Based Emotion Recognition: CNN-Bi-LSTM with Attention Mechanism

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

A Novel and Powerful Dual-Stream Multi-Level Graph Convolution Network for Emotion Recognition