Abstract:Modality alignment can maintain the consistency of semantics in multi-modal emotion recognition tasks, ensuring that features from different modalities accurately represent the emotion-related information in an encoding space. However, current alignment models either focus only on the local fusion of different modal representations or lack a mining process for unimodal specificity information. We design a Semantic Alignment network based on Multi-Spatial learning (SAMS) for multi-modal emotion recognition, which achieves local and global alignment between modalities using high-level emotion representations of different modalities as supervisory signals. SAMS builds a multi-spatial learning framework for each modality, and constructs a self-modal interaction module under this framework based on cross-modal semantic learning. SAMS provides two learning spaces for each modality, one to detect the affective information for a specific modality, and the other to learn semantic knowledge from other modalities. Subsequently, the features of these two spaces are aligned in temporal and utterance levels by homologous encoding and different target constraints. Based on the alignment characteristics of these two spaces, a self-modal interaction is built to investigate the fusion representation by exploring the global correlation between the alignment features in unimodal multi-spatial learning. In experiments, our proposed model yields consistent improvements on two standard multi-modal benchmarks, and outperforms state-of-the-art approaches. The code of our SAMS is available at: https://github.com/xiaomi1024/code_SAMS.

Type-Specific Modality Alignment for Multi-Modal Information Extraction

Multi-modal Siamese Network for Entity Alignment

Leveraging Intra-modal and Inter-modal Interaction for Multi-Modal Entity Alignment

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

MCSFF: Multi-modal Consistency and Specificity Fusion Framework for Entity Alignment

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

A Multi-Modal Entity Alignment Method with Inter-Modal Enhancement

Universal Multi-modal Entity Alignment Via Iteratively Fusing Modality Similarity Paths.

Semantic Alignment Network for Multi-modal Emotion Recognition

Multi-Modal Entity Alignment Method Based on Feature Enhancement

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

Semantic-specific multimodal relation learning for sentiment analysis

Text-centric Alignment for Multi-Modality Learning

Towards Semantic Consistency: Dirichlet Energy Driven Robust Multi-Modal Entity Alignment

Multimodal Question Answering for Unified Information Extraction

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

MMIEA: Multi-modal Interaction Entity Alignment Model for Knowledge Graphs.

MAF - A General Matching and Alignment Framework for Multimodal Named Entity Recognition.

IBMEA: Exploring Variational Information Bottleneck for Multi-modal Entity Alignment

Multi-layer cross-modality attention fusion network for multimodal sentiment analysis