Abstract:Multi-vector retrieval models such as ColBERT [Khattab and Zaharia, 2020] allow token-level interactions between queries and documents, and hence achieve state of the art on many information retrieval benchmarks. However, their non-linear scoring function cannot be scaled to millions of documents, necessitating a three-stage process for inference: retrieving initial candidates via token retrieval, accessing all token vectors, and scoring the initial candidate documents. The non-linear scoring function is applied over all token vectors of each candidate document, making the inference process complicated and slow. In this paper, we aim to simplify the multi-vector retrieval by rethinking the role of token retrieval. We present XTR, ConteXtualized Token Retriever, which introduces a simple, yet novel, objective function that encourages the model to retrieve the most important document tokens first. The improvement to token retrieval allows XTR to rank candidates only using the retrieved tokens rather than all tokens in the document, and enables a newly designed scoring stage that is two-to-three orders of magnitude cheaper than that of ColBERT. On the popular BEIR benchmark, XTR advances the state-of-the-art by 2.8 nDCG@10 without any distillation. Detailed analysis confirms our decision to revisit the token retrieval stage, as XTR demonstrates much better recall of the token retrieval stage compared to ColBERT.

What problem does this paper attempt to address?

This paper attempts to solve the efficiency and performance problems of multi - vector retrieval models when dealing with large - scale documents. Specifically, existing multi - vector retrieval models such as ColBERT perform well in information retrieval benchmark tests, but their non - linear scoring functions lead to complex and time - consuming inference processes. This is mainly reflected in three stages: initially obtaining candidate documents through token retrieval, collecting all token vectors, and scoring candidate documents. This three - stage process not only increases the cost of data loading and floating - point operations but also creates a gap between training and inference, resulting in a lower recall rate. To solve these problems, the paper proposes a new method - XTR (Contextualized Token Retriever), aiming to simplify multi - vector retrieval and improve efficiency. The main innovations of XTR include: 1. **Redefining the role of token retrieval**: By introducing a novel objective function, the model is encouraged to prioritize retrieving the most important document tokens. In this way, the score between the query and the document can be calculated without accessing the entire document. 2. **Eliminating the collection stage**: Since only the retrieved tokens are used for scoring, the step of collecting all token vectors is completely omitted, significantly reducing the computational cost. 3. **Improving the scoring stage**: A new scoring stage is designed, which is two to three orders of magnitude cheaper than ColBERT. 4. **Missing similarity imputation**: To deal with relevant tokens that may not be retrieved, a simple method is proposed to estimate the contribution of these missing tokens to the total score. Through these improvements, XTR not only achieves state - of - the - art results on popular benchmarks such as BEIR but also performs well in multiple cross - language retrieval tasks, demonstrating its high efficiency and superior performance in practical applications.

Rethinking the Role of Token Retrieval in Multi-Vector Retrieval

Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling

ESPN: Memory-Efficient Multi-Vector Information Retrieval

Semi-Parametric Retrieval via Binary Token Index

Understanding the Multi-vector Dense Retrieval Models

ColBERT-XM: A Modular Multi-Vector Representation Model for Zero-Shot Multilingual Information Retrieval

BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models

Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently

Comparing Neighbors Together Makes it Easy: Jointly Comparing Multiple Candidates for Efficient and Effective Retrieval

ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction

Composite Re-Ranking for Efficient Document Search with BERT

Learning to Tokenize for Generative Retrieval

Revisiting Neural Retrieval on Accelerators

List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation

Text–video retrieval re-ranking via multi-grained cross attention and frozen image encoders

Efficient Multi-Vector Dense Retrieval Using Bit Vectors

Dense Sparse Retrieval: Using Sparse Language Models for Inference Efficient Dense Retrieval

An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking

TokenBinder: Text-Video Retrieval with One-to-Many Alignment Paradigm

Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models

DynamicRetriever: A Pre-training Model-based IR System with Neither Sparse nor Dense Index