Abstract:There has been a growing interest in multimodal sentiment analysis and emotion recognition in recent years due to its wide range of practical applications. Multiple modalities allow for the integration of complementary information, improving the accuracy and precision of sentiment and emotion recognition tasks. However, working with multiple modalities presents several challenges, including handling data source heterogeneity, fusing information, aligning and synchronizing modalities, and designing effective feature extraction techniques that capture discriminative information from each modality. This paper introduces a novel framework called "Attention-based Multimodal Sentiment Analysis and Emotion Recognition (AMSAER)" to address these challenges. This framework leverages intra-modality discriminative features and inter-modality correlations in visual, audio, and textual modalities. It incorporates an attention mechanism to facilitate sentiment and emotion classification based on visual, textual, and acoustic inputs by emphasizing relevant aspects of the task. The proposed approach employs separate models for each modality to automatically extract discriminative semantic words, image regions, and audio features. A deep hierarchical model is then developed, incorporating intermediate fusion to learn hierarchical correlations between the modalities at bimodal and trimodal levels. Finally, the framework combines four distinct models through decision-level fusion to enable multimodal sentiment analysis and emotion recognition. The effectiveness of the proposed framework is demonstrated through extensive experiments conducted on the publicly available Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset. The results confirm a notable performance improvement compared to state-of-the-art methods, attaining 85% and 93% accuracy for sentiment analysis and emotion classification, respectively. Additionally, when considering class-wise accuracy, the results indicate that the "angry" emotion and "positive" sentiment are classified more effectively than the other emotions and sentiments, achieving 96.80% and 93.14% accuracy, respectively.

Multi-modal Approach for Affective Computing

Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

A Multimodal Emotion Sensing Platform for Building Emotion-Aware Applications

Multimodal Affect Recognition using Kinect

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Multimodal Characterization of Emotion within Multimedia Space

Modeling And Application Of Multimodal Affective User Interface With Multimedia

End-to-end facial and physiological model for Affective Computing and applications

Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

Novel multimodal emotion detection method using Electroencephalogram and Electrocardiogram signals

Unsupervised multi-modal representation learning for affective computing with multi-corpus wearable data

A Multimodal Dataset for Mixed Emotion Recognition

Impact of Affective Multimedia Content on the Electroencephalogram and Facial Expressions

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

Multimodal emotion recognition based on the fusion of vision, EEG, ECG, and EMG signals