Abstract:Multimodal sentiment classification is a notable research field that aims to refine sentimental information and classify the sentiment tendency from sequential multimodal data. Most existing sentimental recognition algorithms explore multimodal fusion schemes that achieve good performance. However, there are two key challenges to overcome. First, it is essential to effectively extract inter- and intra-modality features prior to fusion, while simultaneously reducing ambiguity. The second challenge is how to learn modality-invariant representations that capture the underlying similarities. In this paper, we present a modality-invariant temporal learning technique and a new gated inter-modality attention mechanism to overcome these issues. For the first challenge, our proposed gated inter-modality attention mechanism performs modality interactions and filters inconsistencies from multiple modalities in an adaptive manner. We also use parallel structures to learn more comprehensive sentimental information in pairs (i.e., acoustic and visual). In addition, to address the second problem, we treat each modality as a multivariate Gaussian distribution (considering each timestamp as a single Gaussian distribution) and use the KL divergence to capture the implicit temporal distribution-level similarities. These strategies are helpful in reducing domain shifts between different modalities and extracting effective sequential modality-invariant representations. We have conducted experiments on several public datasets (i.e., YouTube and MOUD) and the results show that our proposed method outperforms the state-of-the-art multimodal sentiment categorization methods.

Towards a Universal Sentiment Classifier in Multiple Languages

Learning Bilingual Embedding Model for Cross-Language Sentiment Classification

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Joint Bilingual Sentiment Classification with Unlabeled Parallel Corpora.

A Mixture Language Model for the Classification of Chinese Online Reviews.

Cross-Lingual Mixture Model for Sentiment Classification.

A Comparative Study of Cross-Lingual Sentiment Classification

Cross-Lingual Propagation for Deep Sentiment Analysis

Feature subsumption for sentiment classification in multiple languages

Bilingual co-training for sentiment classification of chinese product reviews

The Model Arena for Cross-lingual Sentiment Analysis: A Comparative Study in the Era of Large Language Models

Co-training for Cross-Lingual Sentiment Classification

Emoji-Powered Representation Learning for Cross-Lingual Sentiment Classification

Cross-Lingual Sentiment Classification with Bilingual Document Representation Learning

Deep Learning Paradigm with Transformed Monolingual Word Embeddings for Multilingual Sentiment Analysis

Towards a Unified End-to-End Approach for Fully Unsupervised Cross-Lingual Sentiment Analysis.

Using Bilingual Knowledge and Ensemble Techniques for Unsupervised Chinese Sentiment Analysis.

Collaboratively Training Sentiment Classifiers for Multiple Domains

A Parallel Two-Channel Emotion Classification Method for Chinese Text

Multi-Domain Sentiment Classification Based on Domain-Aware Embedding and Attention

Application of Deep Learning-Based Natural Language Processing in Multilingual Sentiment Analysis