Abstract:Multi‐modal sentiment analysis (MSA) is increasingly becoming a hotspot because it extends the conventional Sentiment analysis (SA) based on texts to multi‐modal content which can provide richer affective information. However, compared with textbased sentiment analysis, multi‐modal sentiment analysis has much more challenges, because the joint learning process on multi‐modal data requires both fine‐grained semantic matching and effective heterogeneous feature fusion. Existing approaches generally infer sentiment type from splicing features extracted from different modalities but neglect the strong semantic correlation among cooccurrence data of different modalities. To solve the challenges, a multi‐level deep correlative network for multimodal sentiment analysis is proposed, which can reduce the semantic gap by analyzing simultaneously the middlelevel semantic features of images and the hierarchical deep correlations. First, the most relevant cross‐modal feature representation is generated with Multi‐modal Deep and discriminative correlation analysis (Multi‐DDCA) while keeping those respective modal feature representations to be discriminative. Second, the high‐level semantic outputs from multi‐modal deep and discriminative correlation analysis are encoded into attention‐correlation cross‐modal feature representation through a co‐attention‐based multimodal correlation submodel, and then they are further merged by multi‐layer neural network to train a sentiment classifier for predicting sentimental categories. Extensive experimental results on five datasets demonstrate the effectiveness of the designed approach, which outperforms several state‐of‐the‐art fusion strategies for sentiment analysis.

MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis

A Deep Multi-Level Attentive network for Multimodal Sentiment Analysis

Multi-layer cross-modality attention fusion network for multimodal sentiment analysis

Various syncretic co‐attention network for multimodal sentiment analysis

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

MASANet: Multi-Aspect Semantic Auxiliary Network for Visual Sentiment Analysis

MSNet: A Deep Architecture Using Multi-Sentiment Semantics for Sentiment-Aware Image Style Transfer

A text guided multi-task learning network for multimodal sentiment analysis

Exploring Multimodal Multiscale Features for Sentiment Analysis Using Fuzzy-Deep Neural Network Learning

Cross-modal Enhancement Network for Multimodal Sentiment Analysis

Multi‐level Deep Correlative Networks for Multi‐modal Sentiment Analysis

Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks.

Multi-level Attention Map Network for Multimodal Sentiment Analysis

SentiNet: Mining Visual Sentiment from Scratch.

Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis

Analyzing multimodal public sentiment based on hierarchical semantic attentional network

Multi-Feature Fusion for Multimodal Attentive Sentiment Analysis.

Deep Learning-Based Natural Language Processing Methods for Sentiment Analysis in Social Networks

Visual-textual Sentiment Classification with Bi-Directional Multi-Level Attention Networks

Transformer-Based Interactive Multi-Modal Attention Network for Video Sentiment Detection

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement