Abstract:State-of-the-art retrieval models typically address a straightforward search scenario, where retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but underperforms a smaller CLIP retriever in cross-modal retrieval tasks due to modality bias from MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose to continually fine-tune the universal multimodal retriever to enhance its text retrieval capability while maintaining multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on MTEB retrieval benchmark. Finally, we explore to prompt the off-the-shelf MLLMs as the zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way to advance universal multimodal retrieval in the future.

Aligning Language Models for Versatile Text-based Item Retrieval

LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding

Enhancing Tool Retrieval with Iterative Feedback from Large Language Models

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

Building an Efficient and Effective Retrieval-based Dialogue System via Mutual Learning

Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

Towards Robust Text Retrieval with Progressive Learning

Bridging Language and Items for Retrieval and Recommendation

Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models

Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation

Item-Language Model for Conversational Recommendation

LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

Retrieve Anything To Augment Large Language Models

BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers

Bridging Items and Language: A Transition Paradigm for Large Language Model-Based Recommendation

Ada-Retrieval: An Adaptive Multi-Round Retrieval Paradigm for Sequential Recommendations

Complement Lexical Retrieval Model with Semantic Residual Embeddings

Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders

ColBERT-XM: A Modular Multi-Vector Representation Model for Zero-Shot Multilingual Information Retrieval

Bridging Language Gaps in Audio-Text Retrieval