Abstract:Multimodal Entity Linking (MEL) is a crucial task that aims at linking ambiguous mentions within multimodal contexts to the referent entities in a multimodal knowledge base, such as Wikipedia. Existing methods focus heavily on using complex mechanisms and extensive model tuning methods to model the multimodal interaction on specific datasets. However, these methods overcomplicate the MEL task and overlook the visual semantic information, which makes them costly and hard to scale. Moreover, these methods can not solve the issues like textual ambiguity, redundancy, and noisy images, which severely degrade their performance. Fortunately, the advent of Large Language Models (LLMs) with robust capabilities in text understanding and reasoning, particularly Multimodal Large Language Models (MLLMs) that can process multimodal inputs, provides new insights into addressing this challenge. However, how to design a universally applicable LLMs-based MEL approach remains a pressing challenge. To this end, we propose UniMEL, a unified framework which establishes a new paradigm to process multimodal entity linking tasks using LLMs. In this framework, we employ LLMs to augment the representation of mentions and entities individually by integrating textual and visual information and refining textual information. Subsequently, we employ the embedding-based method for retrieving and re-ranking candidate entities. Then, with only ~0.26% of the model parameters fine-tuned, LLMs can make the final selection from the candidate entities. Extensive experiments on three public benchmark datasets demonstrate that our solution achieves state-of-the-art performance, and ablation studies verify the effectiveness of all modules. Our code is available at <a class="link-external link-https" href="https://github.com/Javkonline/UniMEL" rel="external noopener nofollow">this https URL</a>.

Enrich cross-lingual entity links for online wikis via multi-modal semantic matching

Cross-Lingual Entity Matching for Heterogeneous Online Wikis.

Boosting Collective Entity Linking via Type-Guided Semantic Embedding.

DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking

A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking

Multi-level Matching Network for Multimodal Entity Linking

Entity Linking Supported Multimodal Data: Fusing Text and Image features for Higher Accuracy

Cross Domain Search by Exploiting Wikipedia.

UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models

MMEL: A Joint Learning Framework for Multi-Mention Entity Linking.

Multimodal Entity Linking with Mixed Fusion Mechanism.

Neural Cross-Lingual Entity Linking

Multi-Grained Multimodal Interaction Network for Entity Linking

Generative Multimodal Entity Linking

Multimodal Entity Linking with Gated Hierarchical Fusion and Contrastive Training

Multi-modal Siamese Network for Entity Alignment

Faster Zero-shot Multi-modal Entity Linking via Visual-Linguistic Representation

Leveraging Intra-modal and Inter-modal Interaction for Multi-Modal Entity Alignment

WIKIDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types

Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization

Multimodal Entity Linking: A New Dataset and A Baseline