Abstract:Multimodal emotion analysis is an important endeavor in human–computer interaction research, as it enables the accurate identification of an individual's emotional state by simultaneously analyzing text, video, and sound features. Although current emotion recognition algorithms have performed well using multimodal fusion strategies, two key challenges remain. The first challenge is the efficient extraction of modality-invariant and modality-specific features prior to fusion, which requires deep feature interactions between the different modalities. The second challenge concerns the ability to distinguish high-level semantic relations between modality features. To address these issues, we propose a new modality-binding learning framework and redesign the internal structure of the transformer model. Our proposed modality binding learning model addresses the first challenge by incorporating bimodal and trimodal binding mechanisms. These mechanisms handle modality-specific and modality-invariant features, respectively, and facilitate cross-modality interactions. Furthermore, we enhance feature interactions by introducing fine-grained convolution modules in the feedforward and attention layers of the transformer structure. To address the second issue, we introduce CLS and PE feature vectors for modality-invariant and modality-specific features, respectively. We use similarity loss and dissimilarity loss to support model convergence. Experiments on the widely used MOSI and MOSEI datasets show that our proposed method outperforms state-of-the-art multimodal sentiment classification approaches , confirming its effectiveness and superiority. The source code can be found at https://github.com/JackAILab/TMBL .

Cross-modal sentiment analysis based on Transformer and image-text collaborative interaction

TensorFormer: A Tensor-Based Multimodal Transformer for Multimodal Sentiment Analysis and Depression Detection

Image and Text Aspect Level Multimodal Sentiment Classification Model Using Transformer and Multilayer Attention Interaction

Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism

Multimodal Sentiment Analysis Based on Transformer and Low-rank Fusion

Feature Fusion Based on Transformer for Cross-modal Retrieval

Transformer-Based Interactive Multi-Modal Attention Network for Video Sentiment Detection

Multi-layer cross-modality attention fusion network for multimodal sentiment analysis

TEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis

A transformer-encoder-based multimodal multi-attention fusion network for sentiment analysis

Tri-CLT: Learning Tri-Modal Representations with Contrastive Learning and Transformer for Multimodal Sentiment Recognition

TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis

Multi-Feature Fusion Multi-Modal Sentiment Analysis Model Based on Cross-Attention Mechanism

Multimodal Sentiment Analysis Based on a Cross-Modal Multihead Attention Mechanism

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos

Visual-Textual Sentiment Analysis Enhanced by Hierarchical Cross-Modality Interaction

SS-Trans (Single-Stream Transformer for Multimodal Sentiment Analysis and Emotion Recognition): The Emotion Whisperer—A Single-Stream Transformer for Multimodal Sentiment Analysis

Leveraging Vision-Language Pre-Trained Model and Contrastive Learning for Enhanced Multimodal Sentiment Analysis

TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis