Abstract:Multimodal Entity Linking (MEL) is a crucial task that aims at linking ambiguous mentions within multimodal contexts to the referent entities in a multimodal knowledge base, such as Wikipedia. Existing methods focus heavily on using complex mechanisms and extensive model tuning methods to model the multimodal interaction on specific datasets. However, these methods overcomplicate the MEL task and overlook the visual semantic information, which makes them costly and hard to scale. Moreover, these methods can not solve the issues like textual ambiguity, redundancy, and noisy images, which severely degrade their performance. Fortunately, the advent of Large Language Models (LLMs) with robust capabilities in text understanding and reasoning, particularly Multimodal Large Language Models (MLLMs) that can process multimodal inputs, provides new insights into addressing this challenge. However, how to design a universally applicable LLMs-based MEL approach remains a pressing challenge. To this end, we propose UniMEL, a unified framework which establishes a new paradigm to process multimodal entity linking tasks using LLMs. In this framework, we employ LLMs to augment the representation of mentions and entities individually by integrating textual and visual information and refining textual information. Subsequently, we employ the embedding-based method for retrieving and re-ranking candidate entities. Then, with only ~0.26% of the model parameters fine-tuned, LLMs can make the final selection from the candidate entities. Extensive experiments on three public benchmark datasets demonstrate that our solution achieves state-of-the-art performance, and ablation studies verify the effectiveness of all modules. Our code is available at <a class="link-external link-https" href="https://github.com/Javkonline/UniMEL" rel="external noopener nofollow">this https URL</a>.

MELO: An Evaluation Benchmark for Multilingual Entity Linking of Occupations

MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

Multilingual End to End Entity Linking

Multimodal Entity Linking: A New Dataset and A Baseline

MELA: Multilingual Evaluation of Linguistic Acceptability

Entity Linking in the Job Market Domain

Generative Multimodal Entity Linking

Multilingual Autoregressive Entity Linking

BELB: a Biomedical Entity Linking Benchmark

UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models

ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

$M^3EL$: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking

Video Multimodal Entity Linking via Multi-Perspective Enhanced Subgraph Contrastive Network

P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs

Evaluating end-to-end entity linking on domain-specific knowledge bases: Learning about ancient technologies from museum collections

Multilingual bi‐encoder models for biomedical entity linking

M-MELD: A Multilingual Multi-Party Dataset for Emotion Recognition in Conversations

HOSMEL: A Hot-Swappable Modularized Entity Linking Toolkit for Chinese

Multilingual Entity Linking Using Dense Retrieval

LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain

VP-MEL: Visual Prompts Guided Multimodal Entity Linking