An efficient multimodal sentiment analysis in social media using hybrid optimal multi-scale residual attention network

Murugesan, Kanipriya

DOI: https://doi.org/10.1007/s10462-023-10645-7

IF: 9.588

2024-02-06

Artificial Intelligence Review

Abstract:Sentiment analysis is a key component of many social media analysis projects. Additionally, prior research has concentrated on a single modality in particular, such as text descriptions for visual information. In contrast to standard image databases, social images frequently connect to one another, making sentiment analysis challenging. The majority of methods now in use consider different images individually, rendering them useless for interrelated images. We proposed a hybrid Arithmetic Optimization Algorithm- Hunger Games Search (AOA-HGS)-optimized Ensemble Multi-scale Residual Attention Network (EMRA-Net) technique in this paper to explore the modal correlations including texts, audio, social links, and video for more effective multimodal sentiment analysis. The hybrid AOA-HGS technique learns complementary and comprehensive features. The EMRA-Net uses two segments, including Ensemble Attention CNN (EA-CNN) and Three-scale Residual Attention Convolutional Neural Network (TRA-CNN), to analyze the multimodal sentiments. The loss of spatial domain image texture features can be reduced by adding the Wavelet transform to TRA-CNN. The feature-level fusion technique known as EA-CNN is used to combine visual, audio, and textual information. The proposed method performs significantly better than the existing multimodel sentimental analysis techniques of HALCB, HDF, and MMLatch when evaluated using the Multimodal Emotion Lines Dataset (MELD) and EmoryNLP datasets. Also, even though the size of the training set varies, the proposed method outperformed other techniques in terms of recall, accuracy, F score, and precision and takes less time to compute in both datasets.

computer science, artificial intelligence

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve This paper aims to address the issue of multimodal sentiment analysis in social media. Specifically, existing methods mostly focus on sentiment analysis of a single modality (such as text descriptions or visual information) and overlook the interrelationships between different modalities. Images in social media are often interrelated, making sentiment analysis more complex. Most existing methods handle different images separately and cannot effectively process interrelated images. To solve these problems, the authors propose a Hybrid Arithmetic Optimization Algorithm-Hunger Games Search (AOA-HGS) optimized Ensemble Multi-scale Residual Attention Network (EMRA-Net) method. This method can comprehensively consider information from various modalities such as text, audio, and video, thereby achieving more effective multimodal sentiment analysis. Additionally, this method outperforms existing multimodal sentiment analysis techniques on the Multimodal Emotion Lines Dataset (MELD) and EmoryNLP datasets, and it also has a shorter computation time.

An efficient multimodal sentiment analysis in social media using hybrid optimal multi-scale residual attention network

Application of Multimodal Data Fusion Attentive Dual Residual Generative Adversarial Network in Sentiment Recognition and Sentiment Analysis

Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks

Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data

AMSA: Adaptive Multimodal Learning for Sentiment Analysis

Multimodal sentiment analysis leveraging the strength of deep neural networks enhanced by the XGBoost classifier

A soft voting ensemble learning-based approach for multimodal sentiment analysis

Multimodal sentiment analysis based on multiple attention

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

Heterogeneous Hierarchical Fusion Network for Multimodal Sentiment Analysis in Real-World Environments

Attention-based multi-level image and text sentiment analysis

Multimodal sentimental analysis for social media applications: A comprehensive review

GCM-Net: Graph-enhanced Cross-Modal Infusion with a Metaheuristic-Driven Network for Video Sentiment and Emotion Analysis

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

A Deep Multi-Level Attentive network for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis Based on Composite Hierarchical Fusion

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism

Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis

Multi‐level feature optimization and multimodal contextual fusion for sentiment analysis and emotion classification