Abstract:Multimodal entity linking (MEL) task, which aims at resolving ambiguous mentions to a multimodal knowledge graph, has attracted wide attention in recent years. Though large efforts have been made to explore the complementary effect among multiple modalities, however, they may fail to fully absorb the comprehensive expression of abbreviated textual context and implicit visual indication. Even worse, the inevitable noisy data may cause inconsistency of different modalities during the learning process, which severely degenerates the performance. To address the above issues, in this paper, we propose a novel Multi-GraIned Multimodal InteraCtion Network $\textbf{(MIMIC)}$ framework for solving the MEL task. Specifically, the unified inputs of mentions and entities are first encoded by textual/visual encoders separately, to extract global descriptive features and local detailed features. Then, to derive the similarity matching score for each mention-entity pair, we device three interaction units to comprehensively explore the intra-modal interaction and inter-modal fusion among features of entities and mentions. In particular, three modules, namely the Text-based Global-Local interaction Unit (TGLU), Vision-based DuaL interaction Unit (VDLU) and Cross-Modal Fusion-based interaction Unit (CMFU) are designed to capture and integrate the fine-grained representation lying in abbreviated text and implicit visual cues. Afterwards, we introduce a unit-consistency objective function via contrastive learning to avoid inconsistency and model degradation. Experimental results on three public benchmark datasets demonstrate that our solution outperforms various state-of-the-art baselines, and ablation studies verify the effectiveness of designed modules.

Dynamic Interaction Networks for Image-Text Multimodal Learning

Multimodal Image-to-Image Translation via Mutual Information Estimation and Maximization

Multimodal Sentiment Analysis With Image-Text Interaction Network

Image-text interaction graph neural network for image-text sentiment analysis

Advanced Multimodal Deep Learning Architecture for Image-Text Matching

Multi-Grained Multimodal Interaction Network for Entity Linking

Unifying Multimodal Transformer for Bi-directional Image and Text Generation

Research on Optimization of Natural Language Processing Model Based on Multimodal Deep Learning

Giving Text More Imagination Space for Image-text Matching

Cross-modal sentiment analysis based on Transformer and image-text collaborative interaction

Meta-Transformer: A Unified Framework for Multimodal Learning

Bidirectional interactive alignment network for image captioning

Multimodal Dialogue Generation Based on Transformer and Collaborative Attention

Multimodal Image Aesthetic Prediction with Missing Modality

A Multimodal Visual Encoding Model Aided by Introducing Verbal Semantic Information

UNIMO-3: Multi-granularity Interaction for Vision-Language Representation Learning

Image Captioning via Dynamic Path Customization

Multimodal English Teaching Classroom Interaction Based on Artificial Neural Network

Dynamic Graph Message Passing Networks for Visual Recognition

Visual interaction networks: A novel bio-inspired computational model for image classification

Deep Multimodal Network for Multi-Label Classification.