Abstract:Recent advancements in image-text matching have been notable, yet prevailing models predominantly cater to broad queries and struggle with accommodating fine-grained query intention. In this paper, we work towards the \textbf{E}ntity-centric \textbf{I}mage-\textbf{T}ext \textbf{M}atching (EITM), a task that the text and image involve specific entity-related information. The challenge of this task mainly lies in the larger semantic gap in entity association modeling, comparing with the general image-text matching <a class="link-external link-http" href="http://problem.To" rel="external noopener nofollow">this http URL</a> narrow the huge semantic gap between the entity-centric text and the images, we take the fundamental CLIP as the backbone and devise a multimodal attentive contrastive learning framework to tam CLIP to adapt EITM problem, developing a model named EntityCLIP. The key of our multimodal attentive contrastive learning is to generate interpretive explanation text using Large Language Models (LLMs) as the bridge clues. In specific, we proceed by extracting explanatory text from off-the-shelf LLMs. This explanation text, coupled with the image and text, is then input into our specially crafted Multimodal Attentive Experts (MMAE) module, which effectively integrates explanation texts to narrow the gap of the entity-related text and image in a shared semantic space. Building on the enriched features derived from MMAE, we further design an effective Gated Integrative Image-text Matching (GI-ITM) strategy. The GI-ITM employs an adaptive gating mechanism to aggregate MMAE's features, subsequently applying image-text matching constraints to steer the alignment between the text and the image. Extensive experiments are conducted on three social media news benchmarks including N24News, VisualNews, and GoodNews, the results shows that our method surpasses the competition methods with a clear margin.

CTextEM: Employing Compound Textual Information in Entity Matching

CTextEM: Using Consolidated Textual Data for Entity Matching

Crowd-Guided Entity Matching with Consolidated Textual Data

Cross-Lingual Entity Matching for Heterogeneous Online Wikis.

Reserch of Entity Matching Based on Multiple Heterogenous Data

Multi-Context Attention for Entity Matching.

MultiEM: Efficient and Effective Unsupervised Multi-Table Entity Matching

EntityCLIP: Entity-Centric Image-Text Matching via Multimodal Attentive Contrastive Learning

Entity Matching Across Heterogeneous Sources

Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching

Topic discovery based on dual EM merging.

Exploring Entity-Level Spatial Relationships for Image-Text Matching

JointMatcher: Numerically-aware entity matching using pre-trained language models with attention concentration

ZeroMatcher: A Cost-Off Entity Matching System

ZeroMatcher

A simple and efficient text matching model based on deep interaction

Effective entity matching with transformers

Heterogeneous Entity Matching with Complex Attribute Associations using BERT and Neural Networks

Entity disambiguation with context awareness in user-generated short texts

Leveraging Large Language Models for Entity Matching

A Technical Report: Entity Extraction Using Both Character-based and Token-based Similarity