Abstract:Multimodal sentiment analysis aims to extract sentiment information expressed by users from multimodal data, including linguistic, acoustic, and visual cues. However, the heterogeneity of multimodal data leads to disparities in modal distribution, thereby impacting the model’s ability to effectively integrate complementarity and redundancy across modalities. Additionally, existing approaches often merge modalities directly after obtaining their representations, overlooking potential emotional correlations between them. To tackle these challenges, we propose a Multiview Collaborative Perception (MVCP) framework for multimodal sentiment analysis. This framework consists primarily of two modules: Multimodal Disentangled Representation Learning (MDRL) and Cross-Modal Context Association Mining (CMCAM). The MDRL module employs a joint learning layer comprising a common encoder and an exclusive encoder. This layer maps multimodal data to a hypersphere, learning common and exclusive representations for each modality, thus mitigating the semantic gap arising from modal heterogeneity. To further bridge semantic gaps and capture complex inter-modal correlations, the CMCAM module utilizes multiple attention mechanisms to mine cross-modal and contextual sentiment associations, yielding joint representations with rich multimodal semantic interactions. In this stage, the CMCAM module only discovers the correlation information among the common representations in order to maintain the exclusive representations of different modalities. Finally, a multitask learning framework is adopted to achieve parameter sharing between single-modal tasks and improve sentiment prediction performance. Experimental results on the MOSI and MOSEI datasets demonstrate the effectiveness of the proposed method.

Multimodal Sentiment Analysis Missing Modality Reconstruction Network Based on Shared-Specific Features

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Multimodal Sentiment Analysis with Missing Modality: A Knowledge-Transfer Approach

Toward Robust Multimodal Learning using Multimodal Foundational Models

Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention

Multimodal Sentiment Analysis based on Supervised Contrastive Learning and Cross-modal Translation under Modalities Missing * .

MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis

Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

Multimodal Sentiment Analysis Based on Disentangled Representation Learning and Cross-Modal-context Association Mining

Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities

Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis

Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities

Towards Robust Multimodal Sentiment Analysis under Uncertain Signal Missing

Modality translation-based multimodal sentiment analysis under uncertain missing modalities

Contrastive Learning based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition with Missing Modalities

FDR-MSA: Enhancing multimodal sentiment analysis through feature disentanglement and reconstruction

Multimodal Sentiment Analysis under Modality Deficiency with Prototype-Augmentation in Software Engineering

Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities

Similar Modality Completion-Based Multimodal Sentiment Analysis under Uncertain Missing Modalities

Learning Speaker-Independent Multimodal Representation for Sentiment Analysis

A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities