Abstract:Image-Text Sentiment Analysis task has garnered increased attention in recent years due to the surge in user-generated content on social media platforms. Previous research efforts have made noteworthy progress by leveraging the affective concepts shared between vision and text modalities. However, emotional cues may reside exclusively within one of the prevailing modalities, owing to modality independent nature and the potential absence of certain modalities. In this study, we aim to emphasize the significance of modality-independent emotional behaviors, in addition to the modality-invariant behaviors. To achieve this, we propose a novel approach called Crossmodal Translation-Based Meta Weight Adaption (CTMWA). Specifically, our approach involves the construction of the crossmodal translation network, which serves as the encoder. This architecture captures the shared concepts between vision content and text, empowering the model to effectively handle scenarios where either the vision or textual modality is missing. Building upon the translation-based framework, we introduce the strategy of unimodal weight adaption. Leveraging the meta-learning paradigm, our proposed strategy gradually learns to acquire unimodal weights for individual instances from a few hand-crafted meta instances with unimodal annotations. This enables us to modulate the gradients of each modality encoder based on the discrepancy between modalities during model training. Extensive experiments are conducted on three benchmark image-text sentiment analysis datasets, namely MVSA-Single, MVSA-Multiple, and TumEmo. The empirical results demonstrate that our proposed approach achieves the highest performance across all conventional image-text databases. Furthermore, experiments under modality missing settings and case study for reliable sentiment prediction are also conducted further exhibiting superior robustness as well as reliability of the propose approach.

Language Model Weight Adaptation Based on Cross-entropy for Statistical Machine Translation

Transductive Minimum Error Rate Training for Statistical Machine Translation.

Adaptation of Language Models for SMT Using Neural Networks with Topic Information.

Translation Model Adaptation for Statistical Machine Translation with Monolingual Topic Information.

Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

Word-based Domain Adaptation for Neural Machine Translation

A Language Model Adaptation Approach Based on Text Classification.

Domain Adaptation for Statistical Machine Translation

Can the Variation of Model Weights be used as a Criterion for Self-Paced Multilingual NMT?

Language Modelling Approaches to Adaptive Machine Translation

Crossmodal Translation Based Meta Weight Adaption for Robust Image-Text Sentiment Analysis

An Investigation On Statistical Machine Translation With Neural Language Models

Learning Domain Specific Sub-layer Latent Variable for Multi-Domain Adaptation Neural Machine Translation

Just-in-time Latent Semantic Adaptation on Language Model for Chinese Speech Recognition Using Web Data

Dynamic Data Selection and Weighting for Iterative Back-Translation

Adaptive Machine Translation with Large Language Models

CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models

Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models

Language Model Adaptation Based on the Classification of a Trigram's Language Style Feature

A New Topic-Based Language Model Adaptation

Adaptive development data selection for log-linear model in statistical machine translation