Abstract:State-of-the-art retrieval models typically address a straightforward search scenario, where retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but underperforms a smaller CLIP retriever in cross-modal retrieval tasks due to modality bias from MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose to continually fine-tune the universal multimodal retriever to enhance its text retrieval capability while maintaining multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on MTEB retrieval benchmark. Finally, we explore to prompt the off-the-shelf MLLMs as the zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way to advance universal multimodal retrieval in the future.

Complement Lexical Retrieval Model with Semantic Residual Embeddings

Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach

Domain-Specific Modeling and Semantic Alignment for Image-Based 3d Model Retrieval

LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding

Semantic-enhanced discriminative embedding learning for cross-modal retrieval

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

Retrieval-Enhanced Machine Learning: Synthesis and Opportunities

A Framework for Image Text Retrieval Based on Large Language Model

Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models

More Room for Language: Investigating the Effect of Retrieval on Language Models

Large Language Models are Strong Zero-Shot Retriever

Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

Aligning Language Models for Versatile Text-based Item Retrieval

A Comparison Between Term-Based and Embedding-Based Methods for Initial Retrieval

Sparse, Dense, and Attentional Representations for Text Retrieval

UnifieR: A Unified Retriever for Large-Scale Retrieval

Complementarity is the king: Multi-modal and multi-grained hierarchical semantic enhancement network for cross-modal retrieval

Retrieve Anything To Augment Large Language Models

Beyond Precision: A Study on Recall of Initial Retrieval with Neural Representations

Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval

Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking