Abstract:State-of-the-art retrieval models typically address a straightforward search scenario, where retrieval tasks are fixed (e.g., finding a passage to answer a specific question) and only a single modality is supported for both queries and retrieved results. This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs), enabling a broader search scenario, termed universal multimodal retrieval, where multiple modalities and diverse retrieval tasks are accommodated. To this end, we first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. Our empirical results show that the fine-tuned MLLM retriever is capable of understanding challenging queries, composed of both text and image, but underperforms a smaller CLIP retriever in cross-modal retrieval tasks due to modality bias from MLLMs. To address the issue, we propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers. Second, we propose to continually fine-tune the universal multimodal retriever to enhance its text retrieval capability while maintaining multimodal retrieval capability. As a result, our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR, which spans multiple domains and tasks, while also surpassing the state-of-the-art text retrieval model, NV-Embed-v1, on MTEB retrieval benchmark. Finally, we explore to prompt the off-the-shelf MLLMs as the zero-shot rerankers to refine the ranking of the candidates from the multimodal retriever. We find that through prompt-and-reranking, MLLMs can further improve multimodal retrieval when the user queries (e.g., text-image composed queries) are more complex and challenging to understand. These findings also pave the way to advance universal multimodal retrieval in the future.

Multi-Modal Retrieval for Multimedia Digital Libraries: Issues, Architecture, and Mechanisms

Search for Multi-modality Data in Digital Libraries

A MULTI-CHANNEL RETRIEVAL SYSTEM FOR MULTIMEDIA DOCUMENTS

Multistage and Multi-features Medical Image Retrieval System

Multiple Kernel Visual-Auditory Representation Learning for Retrieval

The Research of Multimedia Cross Reference Retrieval System

Personalized Multimedia Retrieval In Cadal Digital Library

Multimedia Analysis and Retrieval System

Effective Deep Learning-Based Multi-Modal Retrieval

OCTOPUS: aggressive search of multi-modality data using multifaceted knowledge base.

Understanding Multimedia Document Semantics for Cross-Media Retrieval

Framework of content-based multimedia retrieval for digital library

Cross-Media Retrieval: Concepts, Advances And Challenges

Manifold Learning Based Cross-media Retrieval: A Solution to Media Object Complementary Nature

Multi-Modal Image Retrieval for Complex Queries using Small Codes

Multi-modal Hashing for Efficient Multimedia Retrieval: A Survey

Multimedia information retrieval at a crossroad

Boosting cross-media retrieval by learning with positive and negative examples

Video retrieval with multi-modal features.

Cross-media retrieval using query dependent search methods

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs