Abstract:State-of-the-art retrieval models typically address a straightforward search scenario, where retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but underperforms a smaller CLIP retriever in cross-modal retrieval tasks due to modality bias from MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose to continually fine-tune the universal multimodal retriever to enhance its text retrieval capability while maintaining multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on MTEB retrieval benchmark. Finally, we explore to prompt the off-the-shelf MLLMs as the zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way to advance universal multimodal retrieval in the future.

LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding

Retrieve Anything To Augment Large Language Models

Retrieval-Augmented Retrieval: Large Language Models Are Strong Zero-Shot Retriever.

Making Large Language Models A Better Foundation For Dense Retrieval

A Multi-Task Embedder For Retrieval Augmented LLMs

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models

Large Language Models are Strong Zero-Shot Retriever

Bridging the Preference Gap between Retrievers and LLMs

Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval

Redefining Information Retrieval of Structured Database via Large Language Models

Improving Retrieval-Augmented Large Language Models via Data Importance Learning

Query Rewriting for Retrieval-Augmented Large Language Models

Leveraging LLMs for Unsupervised Dense Retriever Ranking

LLMs are Also Effective Embedding Models: An In-depth Overview

RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit

Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In

Reliable, adaptable, and attributable language models with retrieval

Query Rewriting in Retrieval-Augmented Large Language Models.