Abstract:Microblog sentiment prediction has attracted extensive research focus with wide application prospects. With the increasing proportion of multimodal tweets consisting of images, texts, and emoticons, new challenges have been raised to the existing sentiment prediction schemes. More crucially, it remains an open problem to model the dependency among multiple modalities, where one or more modalities may be missing. In this paper, we present a novel Bi-layer Multimodal Hypergraph learning (Bi-MHG) toward robust sentiment prediction of multimodal tweets to tackle the above challenges. In particular, we design a two-layer structure for the proposed Bi-MHG model: The first layer, that is, a tweet-level hypergraph, learns the tweet-feature correlation and the tweet relevance to predict the sentiments of unlabeled tweets. The second layer, that is, a feature-level hypergraph learns the relevance among different feature modalities (including the midlevel visual features in Sentibank [1]) by leveraging prior multimodal sentiment dictionaries. These two layers are connected by sharing the relevance of multimodal features in a unified bilayer learning scheme. In such a way, Bi-MHG explicitly models the modality relevance rather than implicitly weighting multimodal features adopted in the existing Multimodal Hypergraph learning [2]. Finally, a nested alternating optimization is further proposed for Bi-MHG parameter learning. We have carried out extensive evaluations on a real-world microblog dataset crawled from Sina Weibo. For the task of multimodal sentiment prediction, superior performance is reported over several state-of-the-art and alternative approaches, which demonstrates the merits of the proposed scheme.

Predicting Microblog Sentiments Via Weakly Supervised Multimodal Deep Learning.

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Cross-Modality Microblog Sentiment Prediction Via Bi-Layer Multimodal Hypergraph Learning

Word Dictionary Emoticon Dictionary SentiBank : ANP Detector Library Microblog with labeled sentiment Testing microblogs Update Update W Update g

A Fine-Grained Modal Label-Based Multi-Stage Network for Multimodal Sentiment Analysis.

Multimodal Hypergraph Learning For Microblog Sentiment Prediction

Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis

Towards Robust Multimodal Sentiment Analysis with Incomplete Data

Learning Speaker-Independent Multimodal Representation for Sentiment Analysis

Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention

Microblog Sentiment Classification with Heterogeneous Sentiment Knowledge

A Multi-dimension and Multi-granularity Feature Fusion Method for Chinese Microblog Sentiment Classification

Every Term Has Sentiment: Learning from Emoticon Evidences for Chinese Microblog Sentiment Analysis

A multimodal sentiment recognition method based on attention mechanism

A Joint Model for Chinese Microblog Sentiment Analysis.

A Sentiment Classification Method of Web Social Media Based on Multidimensional and Multilevel Modeling

Dynamic Weighted Multitask Learning and Contrastive Learning for Multimodal Sentiment Analysis

Do Photos Help Express Our Feelings: Incorporating Multimodal Features into Microblog Sentiment Analysis.

An Autoencoder-based Self-Supervised Learning for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning

Sentiment Analysis for Images on Microblogging by Integrating Textual Information with Multiple Kernel Learning.