Abstract:Sentiment analysis and emotion recognition in videos are challenging tasks, given the diversity and complexity of the information conveyed in different modalities. Developing a highly competent framework that effectively addresses the distinct characteristics across various modalities is a primary concern in this domain. Previous studies on combined multimodal sentiment and emotion analysis often overlooked effective fusion for modality integration, intermodal contextual congruity, optimizing concatenated feature spaces, leading to suboptimal architecture. This paper presents a novel framework that leverages the multi-modal contextual information from utterances and applies metaheuristic algorithms to learn the contributing features for utterance-level sentiment and emotion prediction. Our Graph-enhanced Cross-Modal Infusion with a Metaheuristic-Driven Network (GCM-Net) integrates graph sampling and aggregation to recalibrate the modality features for video sentiment and emotion prediction. GCM-Net includes a cross-modal attention module determining intermodal interactions and utterance relevance. A harmonic optimization module employing a metaheuristic algorithm combines attended features, allowing for handling both single and multi-utterance inputs. To show the effectiveness of our approach, we have conducted extensive evaluations on three prominent multi-modal benchmark datasets, CMU MOSI, CMU MOSEI, and IEMOCAP. The experimental results demonstrate the efficacy of our proposed approach, showcasing accuracies of 91.56% and 86.95% for sentiment analysis on MOSI and MOSEI datasets. We have performed emotion analysis for the IEMOCAP dataset procuring an accuracy of 85.66% which signifies substantial performance enhancements over existing methods.

CiteNet: Cross-modal incongruity perception network for multimodal sentiment prediction

Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis

CTHFNet: contrastive translation and hierarchical fusion network for text–video–audio sentiment analysis

Various syncretic co‐attention network for multimodal sentiment analysis

ModalNet: an aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional network

Cross-modal Enhancement Network for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis With Image-Text Interaction Network

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Multimodal Multi-loss Fusion Network for Sentiment Analysis

Interactive Fusion Network with Recurrent Attention for Multimodal Aspect-based Sentiment Analysis.

MSFNet: modality smoothing fusion network for multimodal aspect-based sentiment analysis

A transformer-encoder-based multimodal multi-attention fusion network for sentiment analysis

Multi-Channel Attentive Graph Convolutional Network with Sentiment Fusion for Multimodal Sentiment Analysis

Multimodal sentiment analysis based on cross-instance graph neural networks

Multi-Grained Multimodal Interaction Network for Sentiment Analysis.

NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis

Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition

TeFNA: Text-centered Fusion Network with crossmodal Attention for multimodal sentiment analysis

GCM-Net: Graph-enhanced Cross-Modal Infusion with a Metaheuristic-Driven Network for Video Sentiment and Emotion Analysis