Abstract:In recent years multimodal sentiment analysis (MSA) has been devoted to developing effective fusion mechanisms and has made advances, however, there are several challenges that have not been addressed adequately: the models make insufficient use of important information (inter-modal relevance and independence information) resulting in additional noise, and the traditional ternary symmetric architecture cannot well solve the problem of uneven distribution of task-related information among modalities. Thus, we propose Mutual Information Maximization and Feature Space Separation and Bi-Bimodal Modality Fusion (MFSBF)framework which effectively alleviates these problems. To alleviate the problem of underutilization of important information among modalities, a mutual information maximization module and a feature space separation module have been designed. The mutual information module maximizes the mutual information between two modalities to retain more relevance (modality-invariant) information, while the feature separation module separates fusion features to prevent the loss of independence(modality-specific) information during the fusion process. As different modalities contribute differently to the model, a bimodal fusion architecture is used, which involves the fusion of two bimodal pairs. The architecture focuses more on the modality that contains more task-ralated information and alleviates the problem of uneven distribution of useful information among modalities. The experiment results of our model on two publicly available datasets (CUM-MOSI and CUM-MOSEI) achieved better or comparable results than previous models, which demonstrate the efficacy of our method.

Optimal Multimodal Fusion for Multimedia Data Analysis

Multimodal fusion for multimedia analysis: a survey

Multimodal information fusion for selected multimedia applications

Analysis of Multimodal Data Fusion from an Information Theory Perspective

Multi-modal Deep Analysis for Multimedia

Dynamic Multimodal Fusion in Video Search

Deep Multimodal Data Fusion

Dual Low-Rank Multimodal Fusion

Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects

Tri-Modalities Fusion for Multimodal Sentiment Analysis

Extracting Multimedia Semantics Based On Independent Modality Discovering And Fusion

Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

Effective Techniques for Multimodal Data Fusion: A Comparative Analysis

Bi-attention Modal Separation Network for Multimodal Video Fusion

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

Multimodal Fusion on Low-quality Data: A Comprehensive Survey

Two-Level Multimodal Fusion for Sentiment Analysis in Public Security

A multimodal framework for unsupervised feature fusion

Sparsely Multimodal Data Fusion

Mutual information maximization and feature space separation and bi-bimodal mo-dality fusion for multimodal sentiment analysis