[Spinal cord injuries. An intact nerve can be enough for a successful phrenic nerve stimulation].

A. Karlsson,L. Dernevik,B. Houltz

2009-03-11

Abstract:

What problem does this paper attempt to address?

UsbVisdaNet: User Behavior Visual Distillation and Attention Network for Multimodal Sentiment Classification

Shangwu Hou,Gulanbaier Tuerhong,Mairidan Wushouer

DOI: https://doi.org/10.3390/s23104829

2023-05-17

Abstract:In sentiment analysis, biased user reviews can have a detrimental impact on a company's evaluation. Therefore, identifying such users can be highly beneficial as their reviews are not based on reality but on their characteristics rooted in their psychology. Furthermore, biased users may be seen as instigators of other prejudiced information on social media. Thus, proposing a method to help detect polarized opinions in product reviews would offer significant advantages. This paper proposes a new method for sentiment classification of multimodal data, which is called UsbVisdaNet (User Behavior Visual Distillation and Attention Network). The method aims to identify biased user reviews by analyzing their psychological behaviors. It can identify both positive and negative users and improves sentiment classification results that may be skewed due to subjective biases in user opinions by leveraging user behavior information. Through ablation and comparison experiments, the effectiveness of UsbVisdaNet is demonstrated, achieving superior sentiment classification performance on the Yelp multimodal dataset. Our research pioneers the integration of user behavior features, text features, and image features at multiple hierarchical levels within this domain.
Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Hao Sun,Jiaqing Liu,Yen-Wei Chen,Lanfen Lin

DOI: https://doi.org/10.1016/j.inffus.2022.10.031

IF: 18.6

2023-01-01

Information Fusion

Abstract:Multimodal sentiment classification is a notable research field that aims to refine sentimental information and classify the sentiment tendency from sequential multimodal data. Most existing sentimental recognition algorithms explore multimodal fusion schemes that achieve good performance. However, there are two key challenges to overcome. First, it is essential to effectively extract inter- and intra-modality features prior to fusion, while simultaneously reducing ambiguity. The second challenge is how to learn modality-invariant representations that capture the underlying similarities. In this paper, we present a modality-invariant temporal learning technique and a new gated inter-modality attention mechanism to overcome these issues. For the first challenge, our proposed gated inter-modality attention mechanism performs modality interactions and filters inconsistencies from multiple modalities in an adaptive manner. We also use parallel structures to learn more comprehensive sentimental information in pairs (i.e., acoustic and visual). In addition, to address the second problem, we treat each modality as a multivariate Gaussian distribution (considering each timestamp as a single Gaussian distribution) and use the KL divergence to capture the implicit temporal distribution-level similarities. These strategies are helpful in reducing domain shifts between different modalities and extracting effective sequential modality-invariant representations. We have conducted experiments on several public datasets (i.e., YouTube and MOUD) and the results show that our proposed method outperforms the state-of-the-art multimodal sentiment categorization methods.
ModalNet: an aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional network

Zhe Zhang,Zhu Wang,Xiaona Li,Nannan Liu,Bin Guo,Zhiwen Yu

DOI: https://doi.org/10.1007/s11280-021-00955-7

2021-09-20

World Wide Web

Abstract:Aspect-level sentiment classification aims to identify sentiment polarity over each aspect of a sentence. In the past, such analysis tasks mainly relied on text data. Nowadays, due to the popularization of smart devices and Internet services, people are generating more abundant data, including text, image, video, et al. Multimodal data from the same post (e.g., a tweet) usually has certain correlation. For example, image data might has an auxiliary effect on the text data, and reasonable processing of such multimodal data can help obtain much richer information for sentiment analysis. To this end, we propose an aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional network. Specifically, we first leverage two memory networks for mining the intra-modality information of text and image, and then design a discriminant matrix to supervise the fusion of inter-modality information. Experimental results demonstrate the effectiveness of the proposed model.
Gated attention fusion network for multimodal sentiment classification

Yongping Du,Yang Liu,Zhi Peng,Xingnan Jin

DOI: https://doi.org/10.1016/j.knosys.2021.108107

2022-03-01

Abstract:Sentiment classification can explore the opinions expressed by people and help them make better decisions. With the increasing of multimodal contents on the web, such as text, image, audio and video, how to make full use of them is important in many tasks, including sentiment classification. This paper focuses on the text and image. Previous work cannot capture the fine-grained features of images, and those models bring a lot of noise during feature fusion. In this work, we propose a novel multimodal sentiment classification model based on gated attention mechanism. The image feature is used to emphasize the text segment by the attention mechanism and it allows the model to focus on the text that affects the sentiment polarity. Moreover, the gating mechanism enables the model to retain useful image information while ignoring the noise introduced during the fusion of image and text. The experiment results on Yelp multimodal dataset show that our model outperforms the previous SOTA model. And the ablation experiment results further prove the effectiveness of different strategies in the proposed model.

computer science, artificial intelligence
Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis

Feiran Huang,Kaimin Wei,Jian Weng,Zhoujun Li

DOI: https://doi.org/10.1145/3388861

2020-08-31

Abstract:Sentiment analysis of social multimedia data has attracted extensive research interest and has been applied to many tasks, such as election prediction and products evaluation. Sentiment analysis of one modality (e.g., text or image) has been broadly studied. However, not much attention has been paid to the sentiment analysis of multimodal data. Different modalities usually have information that is complementary. Thus, it is necessary to learn the overall sentiment by combining the visual content with text description. In this article, we propose a novel method—Attention-Based Modality-Gated Networks (AMGN)—to exploit the correlation between the modalities of images and texts and extract the discriminative features for multimodal sentiment analysis. Specifically, a visual-semantic attention model is proposed to learn attended visual features for each word. To effectively combine the sentiment information on the two modalities of image and text, a modality-gated LSTM is proposed to learn the multimodal features by adaptively selecting the modality that presents stronger sentiment information. Then a semantic self-attention model is proposed to automatically focus on the discriminative features for sentiment classification. Extensive experiments have been conducted on both manually annotated and machine weakly labeled datasets. The results demonstrate the superiority of our approach through comparison with state-of-the-art models.
A Deep Multi-Level Attentive network for Multimodal Sentiment Analysis

Ashima Yadav,Dinesh Kumar Vishwakarma

DOI: https://doi.org/10.1145/3517139

2022-03-16

Abstract:Multimodal sentiment analysis has attracted increasing attention with broad application prospects. Most of the existing methods have focused on a single modality, which fails to handle social media data due to its multiple modalities. Moreover, in multimodal learning, most of the works have focused on simply combining the two modalities without exploring the complicated correlations between them. This resulted in dissatisfying performance for multimodal sentiment classification. Motivated by the status quo, we propose a Deep Multi-level Attentive network (DMLANet), which exploits the correlation between image and text modalities to improve multimodal learning. Specifically, we generate the bi-attentive visual map along the spatial and channel dimensions to magnify Convolutional neural network representation power. Then we model the correlation between the image regions and semantics of the word by extracting the textual features related to the bi-attentive visual features by applying semantic attention. Finally, self-attention is employed to fetch the sentiment-rich multimodal features for the classification automatically. We conduct extensive evaluations on four real-world datasets, namely, MVSA-Single, MVSA-Multiple, Flickr, and Getty Images, which verify our method's superiority.
Social Image-text Sentiment Classification With Cross-Modal Consistency and Knowledge Distillation

Huan Liu,Ke Li,Jianping Fan,Caixia Yan,Tao Qin,Qinghua Zheng

DOI: https://doi.org/10.1109/taffc.2022.3220762

IF: 13.99

2022-01-01

IEEE Transactions on Affective Computing

Abstract:Social media sentiment analysis, which aims to evaluate the attitudes of online users based on their posts, has attracted significant research attention due to its successful application in the field of social media monitoring. It is a beneficial way to utilize multimodal information uploaded by users in order to improve sentiment classification ability. However, existing multimodal fusion-based approaches continue to face difficulties due to the issues of between-modality semantic inconsistency and missing modality. To address these issues, we propose a cross-modal consistency modeling-based knowledge distillation framework for image-text sentiment classification of social media data. Specifically, we design a hybrid curriculum learning strategy to measure the semantic consistency of multimodal data, then gradually train all image-text pairs from easy to hard, which can effectively handle the massive amounts of noise caused by inconsistencies between image and text data on social media. Moreover, in order to alleviate the problem of missing images in unimodal posts, we propose a privileged feature distillation method, in which the teacher model additionally considers images as privileged features, to transfer the visual knowledge to the student model, thereby enhancing the accuracy for text sentiment classification. Extensive experiments conducted over three real-world social media datasets demonstrate the effectiveness and superiority of the proposed multimodal sentiment analysis model.

computer science, cybernetics, artificial intelligence
Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

Sun Zhang,Bo Li,Chunyong Yin

DOI: https://doi.org/10.3390/s22010074

IF: 3.9

2021-12-23

Sensors

Abstract:The rising use of online media has changed the social customs of the public. Users have become accustomed to sharing daily experiences and publishing personal opinions on social networks. Social data carrying emotion and attitude has provided significant decision support for numerous tasks in sentiment analysis. Conventional methods for sentiment classification only concern textual modality and are vulnerable to the multimodal scenario, while common multimodal approaches only focus on the interactive relationship among modalities without considering unique intra-modal information. A hybrid fusion network is proposed in this paper to capture both inter-modal and intra-modal features. Firstly, in the stage of representation fusion, a multi-head visual attention is proposed to extract accurate semantic and sentimental information from textual contents, with the guidance of visual features. Then, multiple base classifiers are trained to learn independent and diverse discriminative information from different modal representations in the stage of decision fusion. The final decision is determined based on fusing the decision supports from base classifiers via a decision fusion method. To improve the generalization of our hybrid fusion network, a similarity loss is employed to inject decision diversity into the whole model. Empiric results on five multimodal datasets have demonstrated that the proposed model achieves higher accuracy and better generalization capacity for multimodal sentiment analysis.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Various syncretic co‐attention network for multimodal sentiment analysis

Meng Cao,Yonghua Zhu,Wenjing Gao,Mengyao Li,Shaoxiu Wang

DOI: https://doi.org/10.1002/cpe.5954

2020-07-22

Concurrency and Computation: Practice and Experience

Abstract:<p>The multimedia contents shared on social network reveal public sentimental attitudes toward specific events. Therefore, it is necessary to conduct sentiment analysis automatically on abundant multimedia data posted by the public for real‐world applications. However, approaches to single‐modal sentiment analysis neglect the internal connections between textual and visual contents, and current multimodal methods fail to exploit the multilevel semantic relations of heterogeneous features. In this article, the various syncretic co‐attention network is proposed to excavate the intricate multilevel corresponding relations between multimodal data, and combine the unique information of each modality for integrated complementary sentiment classification. Specifically, a multilevel co‐attention module is constructed to explore localized correspondences between each image region and each text word, and holistic correspondences between global visual information and context‐based textual semantics. Then, all the single‐modal features can be fused from different levels, respectively. Except for fused multimodal features, our proposed VSCN also considers unique information of each modality simultaneously and integrates them into an end‐to‐end framework for sentiment analysis. The superior results of experiments on three constructed real‐world datasets and a benchmark dataset of Visual Sentiment Ontology (VSO) prove the effectiveness of our proposed VSCN. Especially qualitative analyses are given for deep explaining of our method.</p>
Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention

Ting Wu,Junjie Peng,Wenqiang Zhang,Huiran Zhang,Chuanshuai Ma,Yansong Huang

DOI: https://doi.org/10.1016/j.knosys.2021.107676

2021-11-16

Abstract:Humans express feelings or emotions via different channels. Take language as an example, it entails different sentiments under different visual-acoustic contexts. To precisely understand human intentions as well as reduce the misunderstandings caused by ambiguity and sarcasm, we should consider multimodal signals including textual, visual and acoustic signals. The crucial challenge is to fuse different modalities of features for sentiment analysis. To effectively fuse the information carried by different modalities and better predict the sentiments, we design a novel multi-head attention based fusion network, which is inspired by the observations that the interactions between any two pair-wise modalities are different and they do not equally contribute to the final sentiment prediction. By assigning the acoustic-visual, acoustic-textual and visual-textual features with reasonable attention and exploiting a residual structure, we attend to attain the significant features. We conduct extensive experiments on four public multimodal datasets including one in Chinese and three in English. The results show that our approach outperforms the existing methods and can explain the contributions of bimodal interaction in multiple modalities.

Artificial Intelligence
MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis

Nan Xu,W. Mao

DOI: https://doi.org/10.1145/3132847.3133142

2017-11-06

Abstract:With the prevalence of more diverse and multiform user-generated content in social networking sites, multimodal sentiment analysis has become an increasingly important research topic in recent years. Previous work on multimodal sentiment analysis directly extracts feature representation of each modality and fuse these features for classification. Consequently, some detailed semantic information for sentiment analysis and the correlation between image and text have been ignored. In this paper, we propose a deep semantic network, namely MultiSentiNet, for multimodal sentiment analysis. We first identify object and scene as salient detectors to extract deep semantic features of images. We then propose a visual feature guided attention LSTM model to extract words that are important to understand the sentiment of whole tweet and aggregate the representation of those informative words with visual semantic features, object and scene. The experiments on two public available sentiment datasets verify the effectiveness of our MultiSentiNet model and show that our extracted semantic features demonstrate high correlations with human sentiments.

Computer Science
Transformer-Based Interactive Multi-Modal Attention Network for Video Sentiment Detection

Xuqiang Zhuang,Fangai Liu,Jian Hou,Jianhua Hao,Xiaohong Cai

DOI: https://doi.org/10.1007/s11063-021-10713-5

IF: 2.565

2022-01-04

Neural Processing Letters

Abstract:Social media allows users to express opinions in multiple modalities such as text, pictures, and short-videos. Multi-modal sentiment detection can more effectively predict the emotional tendencies expressed by users. Therefore, multi-modal sentiment detection has received extensive attention in recent years. Current works consider utterances from videos as independent modal, ignoring the effective interaction among diffence modalities of a video. To tackle these challenges, we propose transformer-based interactive multi-modal attention network to investigate multi-modal paired attention between multiple modalities and utterances for video sentiment detection. Specifically, we first take a series of utterances as input and use three separate transformer encoders to capture the utterances-level features of each modality. Subsequently, we introduced multimodal paired attention mechanisms to learn the cross-modality information between multiple modalities and utterances. Finally, we inject the cross-modality information into the multi-headed self-attention layer for making final emotion and sentiment classification. Our solutions outperform baseline models on three multi-modal datasets.

computer science, artificial intelligence
Multi-Channel Attentive Graph Convolutional Network with Sentiment Fusion for Multimodal Sentiment Analysis

Luwei Xiao,Xingjiao Wu,Wen Wu,Jing Yang,Liang He

DOI: https://doi.org/10.1109/icassp43922.2022.9747542

2022-05-23

Abstract:Nowadays, with the explosive growth of multimodal reviews on social media platforms, multimodal sentiment analysis has recently gained popularity because of its high relevance to these social media posts. Although most previous studies design various fusion frameworks for learning an interactive representation of multiple modalities, they fail to incorporate sentimental knowledge into inter-modality learning. This pa-per proposes a Multi-channel Attentive Graph Convolutional Network (MAGCN), consisting of two main components: cross-modality interactive learning and sentimental feature fusion. For cross-modality interactive learning, we exploit the self-attention mechanism combined with densely connected graph convolutional networks to learn inter-modality dynamics. For sentimental feature fusion, we utilize multi-head self-attention to merge sentimental knowledge into inter-modality feature representations. Extensive experiments are conducted on three widely-used datasets. The experimental results demonstrate that the proposed model achieves competitive performance on accuracy and F1 scores compared to several state-of-the-art approaches.
Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data

Akshi Kumar,Kathiravan Srinivasan,Wen-Huang Cheng,Albert Y. Zomaya

DOI: https://doi.org/10.1016/j.ipm.2019.102141

2020-01-01

Abstract:Detecting sentiments in natural language is tricky even for humans, making its automated detection more complicated. This research proffers a hybrid deep learning model for fine-grained sentiment prediction in real-time multimodal data. It reinforces the strengths of deep learning nets in combination to machine learning to deal with two specific semiotic systems, namely the textual (written text) and visual (still images) and their combination within the online content using decision level multimodal fusion. The proposed contextual ConvNet-SVMBoVW model, has four modules, namely, the discretization, text analytics, image analytics, and decision module. The input to the model is multimodal text, m ε {text, image, info-graphic}. The discretization module uses Google Lens to separate the text from the image, which is then processed as discrete entities and sent to the respective text analytics and image analytics modules. Text analytics module determines the sentiment using a hybrid of a convolution neural network (ConvNet) enriched with the contextual semantics of SentiCircle. An aggregation scheme is introduced to compute the hybrid polarity. A support vector machine (SVM) classifier trained using bag-of-visual-words (BoVW) for predicting the visual content sentiment. A Boolean decision module with a logical OR operation is augmented to the architecture which validates and categorizes the output on the basis of five fine-grained sentiment categories (truth values), namely 'highly positive,' 'positive,' 'neutral,' 'negative' and 'highly negative.' The accuracy achieved by the proposed model is nearly 91% which is an improvement over the accuracy obtained by the text and image modules individually.

computer science, information systems,information science & library science
Targeted Aspect-Based Multimodal Sentiment Analysis: An Attention Capsule Extraction and Multi-Head Fusion Network

Donghong Gu,Jiaqian Wang,Shaohua Cai,Chi Yang,Zhengxin Song,Haoliang Zhao,Luwei Xiao,Hua Wang

DOI: https://doi.org/10.1109/access.2021.3126782

IF: 3.9

2021-01-01

IEEE Access

Abstract:Multimodal sentiment analysis has currently identified its significance in a variety of domains. For the purpose of sentiment analysis, different aspects of distinguishing modalities, which correspond to one target, are processed and analyzed. In this work, the researchers propose the targeted aspect-based multimodal sentiment analysis (TABMSA) for the first time. Furthermore, an attention capsule extraction and multi-head fusion network (EF-Net) on the task of TABMSA is devised. The multi-head attention (MHA) based network and the ResNet-152 are employed to deal with texts and images, respectively. The integration of MHA and capsule network aims to capture the interaction among the multimodal inputs. In addition to the targeted aspect, the information from the context and the image is also incorporated for sentiment delivered. The researchers evaluate the proposed model on two manually annotated datasets. the experimental results demonstrate the effectiveness of our proposed model for this new task.

computer science, information systems,telecommunications,engineering, electrical & electronic
MSFNet: modality smoothing fusion network for multimodal aspect-based sentiment analysis

Yan Xiang,Yunjia Cai,Junjun Guo

DOI: https://doi.org/10.3389/fphy.2023.1187503

IF: 3.718

2023-05-11

Frontiers in Physics

Abstract:Multimodal aspect-based sentiment classification (MABSC) aims to determine the sentiment polarity of a given aspect in a sentence by combining text and image information. Although the text and the corresponding image in a sample are associated with aspect information, their features are represented in distinct semantic spaces, creating a substantial semantic gap. Previous research focused primarily on identifying and fusing aspect-level sentiment expressions of different modalities while ignoring their semantic gap. To this end, we propose a novel aspect-based sentiment analysis model named modality smoothing fusion network (MSFNet). In this model, we process the unimodal aspect-aware features via the feature smoothing strategy to partially bridge modality gap. Then we fuse the smoothed features deeply using the multi-channel attention mechanism, to obtain aspect-level sentiment representation with comprehensive representing capability, thereby improving the performance of sentiment classification. Experiments on two benchmark datasets, Twitter2015 and Twitter2017, demonstrate that our model outperforms the second-best model by 1.96% and 0.19% in terms of Macro-F1, respectively. Additionally, ablation studies provide evidence supporting the efficacy of each of our proposed modules. We release the code at: https://github.com/YunjiaCai/MSFNet.

physics, multidisciplinary
Multi-Grained Fusion Network with Self-Distillation for Aspect-Based Multimodal Sentiment Analysis

Juan Yang,Yali Xiao,Xu Du

DOI: https://doi.org/10.1016/j.knosys.2024.111724

IF: 8.139

2024-04-01

Knowledge-Based Systems

Abstract:Aspect-based multimodal sentiment analysis (ABMSA) is an important branch of multimodal sentiment analysis. The goal of ABMSA is to use multimodal information to infer users' sentiment polarity toward the targeted aspect for supporting corresponding decision-making. The existing ABMSA methods usually focus on exploring aspect-aware fine-grained interactions and demonstrate the benefits of integrating multimodal information. However, such approaches still suffer from the following limitations: (1) coarse-grained semantic information extraction is ignored, (2) the relevance between image and aspect is ignored, and (3) model-agnostic techniques to improve multimodal representations are not employed. To address these limitations, we propose a novel multimodal approach named multi-grained fusion network with self-distillation (MGFN-SD). This approach includes unimodal feature extraction, multi-grained representation learning, and self-distillation-based sentiment prediction. The multi-grained representation learning module extracts fine- and coarse-grained interactions based on the aspect–image relevance computation and the similarity calculation for dynamically filtering potential noise brought by the original image. In the sentiment prediction module, self-distillation is employed to transfer knowledge from both hard and soft labels to supervise the training process of each student classifier for improving the quality of multimodal representations. The experimental results on three benchmark datasets indicate the superiority, rationality, and robustness of MGFN-SD.

computer science, artificial intelligence
A multimodal sentiment recognition method based on attention mechanism

Jidong Zhang,Yuxiao Xu,Guanzhi Qu,Jianqiang Li,Bo Liu,Yan Pei

DOI: https://doi.org/10.1145/3582099.3582131

2022-12-17

Abstract:Effective sentiment analysis on social media data can help to better understand the public's sentiment and opinion tendencies. Combining multimodal content for sentiment classification uses the correlation information of data between modalities, thereby avoiding the situation that a single modality does not fully grasp the overall sentiment. This paper proposes a multimodal sentiment recognition model based on the attention mechanism. Through transfer learning, the latest pre-trained model is used to extract preliminary features of text and image, and the attention mechanism is deployed to achieve further feature extraction of prominent image key regions and text keywords, better mining the internal information of modalities and learning the interaction between modalities. In view of the different contribution of each modal to sentiment classification, a decision-level fusion method is proposed to design fusion rules to integrate the classification results of each modal to obtain the final sentiment recognition result. This model integrates various unimodal features well, and effectively mines the emotional information expressed in Internet social media comments. This method is experimentally tested on the Twitter dataset, and the results show that the classification accuracy of sentiment recognition is significantly improved compared with the single-modal method.

Computer Science
Image and Text Aspect Level Multimodal Sentiment Classification Model Using Transformer and Multilayer Attention Interaction

Xiuye Yin,Liyong Chen

DOI: https://doi.org/10.4018/ijdwm.333854

2023-11-15

International Journal of Data Warehousing and Mining

Abstract:Many existing image and text sentiment analysis methods only consider the interaction between image and text modalities, while ignoring the inconsistency and correlation of image and text data, to address this issue, an image and text aspect level multimodal sentiment analysis model using transformer and multi-layer attention interaction is proposed. Firstly, ResNet50 is used to extract image features, and RoBERTa-BiLSTM is used to extract text and aspect level features. Then, through the aspect direct interaction mechanism and deep attention interaction mechanism, multi-level fusion of aspect information and graphic information is carried out to remove text and images unrelated to the given aspect. The emotional representations of text data, image data, and aspect type sentiments are concatenated, fused, and fully connected. Finally, the designed sentiment classifier is used to achieve sentiment analysis in terms of images and texts. This effectively has improved the performance of sentiment discrimination in terms of graphics and text.

computer science, software engineering
Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

Jie Xu,Zhoujun Li,Feiran Huang,Chaozhuo Li,Philip S. Yu

DOI: https://doi.org/10.1109/tii.2020.3005405

IF: 12.3

2021-04-01

IEEE Transactions on Industrial Informatics

Abstract:In the circumstance of social big data, sentiment analysis is attracting increasing attention for its capacity in understanding individuals' attitudes and feelings. Traditional sentiment analysis methods focus on single modality and become ineffective as enormous data are emerging on the social websites with multiple manifestations. In this article, multimodal learning approaches are proposed to capture the relations between image and text, which only stay at the region level and ignore the fact that the channels are also closely correlated with the semantic information. In addition, social images in the social platforms are closely connected by various types of relations, which are also conducice to sentiment classification but neglected by most existing works. In this article, we propose an attention-based heterogeneous relational model to improve the multimodal sentiment analysis performance by incorporating rich social information. Specifically, we propose a progressive dual attention module to capture the correlations between image and text, and then learn the joint image-text representation from the perspective of content information. A channel attention schema is proposed here to highlight semantically rich image channels and a region attention schema is further designed to highlight the emotional regions based on the attended channels. After that, we construct a heterogeneous relation network and extend graph convolutional network to aggregate the content information from social contexts as complements to learn high-quality representations of social images. Our proposal is thoroughly evaluated on two benchmark datasets, and experimental results demonstrate the superiority of the proposed model.

automation & control systems,computer science, interdisciplinary applications,engineering, industrial

[Spinal cord injuries. An intact nerve can be enough for a successful phrenic nerve stimulation].

UsbVisdaNet: User Behavior Visual Distillation and Attention Network for Multimodal Sentiment Classification

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

ModalNet: an aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional network

Gated attention fusion network for multimodal sentiment classification

Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis

A Deep Multi-Level Attentive network for Multimodal Sentiment Analysis

Social Image-text Sentiment Classification With Cross-Modal Consistency and Knowledge Distillation

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

Various syncretic co‐attention network for multimodal sentiment analysis

Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention

MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis

Transformer-Based Interactive Multi-Modal Attention Network for Video Sentiment Detection

Multi-Channel Attentive Graph Convolutional Network with Sentiment Fusion for Multimodal Sentiment Analysis

Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data

Targeted Aspect-Based Multimodal Sentiment Analysis: An Attention Capsule Extraction and Multi-Head Fusion Network

MSFNet: modality smoothing fusion network for multimodal aspect-based sentiment analysis

Multi-Grained Fusion Network with Self-Distillation for Aspect-Based Multimodal Sentiment Analysis

A multimodal sentiment recognition method based on attention mechanism

Image and Text Aspect Level Multimodal Sentiment Classification Model Using Transformer and Multilayer Attention Interaction

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations