Abstract:Cross-lingual cross-modal retrieval (CCR) aims to retrieve visually relevant content based on non-English queries, without relying on human-labeled cross-modal data pairs during training. One popular approach involves utilizing machine translation (MT) to create pseudo-parallel data pairs, establishing correspondence between visual and non-English textual data. However, aligning their representations poses challenges due to the significant semantic gap between vision and text, as well as the lower quality of non-English representations caused by pre-trained encoders and data noise. To overcome these challenges, we propose LECCR, a novel solution that incorporates the multi-modal large language model (MLLM) to improve the alignment between visual and non-English representations. Specifically, we first employ MLLM to generate detailed visual content descriptions and aggregate them into multi-view semantic slots that encapsulate different semantics. Then, we take these semantic slots as internal features and leverage them to interact with the visual features. By doing so, we enhance the semantic information within the visual features, narrowing the semantic gap between modalities and generating local visual semantics for subsequent multi-level matching. Additionally, to further enhance the alignment between visual and non-English features, we introduce softened matching under English guidance. This approach provides more comprehensive and reliable inter-modal correspondences between visual and non-English features. Extensive experiments on four CCR benchmarks, \ie Multi30K, MSCOCO, VATEX, and MSR-VTT-CN, demonstrate the effectiveness of our proposed method. Code: \url{<a class="link-external link-https" href="https://github.com/LiJiaBei-7/leccr" rel="external noopener nofollow">this https URL</a>}.

Msra Columbus At Geoclef 2006

MSRA Columbus at GeoCLEF2007.

Exploring LDA-Based Document Model for Geographic Information Retrieval

Mesa: A Map Service To Support Fuzzy Type-Ahead Search Over Geo-Textual Data

Query Parsing Task for GeoCLEF2007 Report.

Domain-specific Cross-Language Relevant Question Retrieval.

CUHK Experiments with ImageCLEF 2005.

MSRA Att TRECVID 2008: High-Level Feature Extraction and Automatic Search.

GeoCLEF 2007: the CLEF 2007 Cross-Language Geographic Information Retrieval Track Overview.

The MSRA Machine Translation System for IWSLT 2010.

CUHK at ImageCLEF 2005: cross-language and cross-media image retrieval

MSRA-USTC-SJTU at TRECVID 2007: High-Level Feature Extraction and Search.

Multi-lingual Geoparsing based on Machine Translation

Research on Expert Search at Enterprise Track of TREC 2005.

GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model

TREC-10 Web Track Experiments at MSRA.

Bilingual query translation and expansion for supporting more effective cross-language image retrieval.

Indexing implicit locations for geographical information retrieval

Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval

GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmark

Global–Local Information Soft-Alignment for Cross-Modal Remote-Sensing Image–Text Retrieval