Abstract:State-of-the-art retrieval models typically address a straightforward search scenario, where retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but underperforms a smaller CLIP retriever in cross-modal retrieval tasks due to modality bias from MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose to continually fine-tune the universal multimodal retriever to enhance its text retrieval capability while maintaining multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on MTEB retrieval benchmark. Finally, we explore to prompt the off-the-shelf MLLMs as the zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way to advance universal multimodal retrieval in the future.

TELLMe: Teaching and Exploiting Large Language Models for Model Selection in Text Retrieval

Fine-Tuning LLaMA for Multi-Stage Text Retrieval

LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models

A Two-Stage Adaptation of Large Language Models for Text Ranking

TeC: A Novel Method for Text Clustering with Large Language Models Guidance and Weakly-Supervised Contrastive Learning

TnT-LLM: Text Mining at Scale with Large Language Models

Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation

T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step

LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

Large Language Model-guided Document Selection

Enhancing Subtask Performance of Multi-modal Large Language Model

TOME: A Two-stage Approach for Model-based Retrieval

RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit

Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation

How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning?

Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models

Towards Robust Text Retrieval with Progressive Learning

TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs

ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models