Abstract:Traditionally, sparse retrieval systems relied on lexical representations to retrieve documents, such as BM25, dominated information retrieval tasks. With the onset of pre-trained transformer models such as BERT, neural sparse retrieval has led to a new paradigm within retrieval. Despite the success, there has been limited software supporting different sparse retrievers running in a unified, common environment. This hinders practitioners from fairly comparing different sparse models and obtaining realistic evaluation results. Another missing piece is, that a majority of prior work evaluates sparse retrieval models on in-domain retrieval, i.e. on a single dataset: MS MARCO. However, a key requirement in practical retrieval systems requires models that can generalize well to unseen out-of-domain, i.e. zero-shot retrieval tasks. In this work, we provide SPRINT, a unified Python toolkit based on Pyserini and Lucene, supporting a common interface for evaluating neural sparse retrieval. The toolkit currently includes five built-in models: uniCOIL, DeepImpact, SPARTA, TILDEv2 and SPLADEv2. Users can also easily add customized models by defining their term weighting method. Using our toolkit, we establish strong and reproducible zero-shot sparse retrieval baselines across the well-acknowledged benchmark, BEIR. Our results demonstrate that SPLADEv2 achieves the best average score of 0.470 nDCG@10 on BEIR amongst all neural sparse retrievers. In this work, we further uncover the reasons behind its performance gain. We show that SPLADEv2 produces sparse representations with a majority of tokens outside of the original query and document which is often crucial for its performance gains, i.e. a limitation among its other sparse counterparts. We provide our SPRINT toolkit, models, and data used in our experiments publicly here at <a class="link-external link-https" href="https://github.com/thakur-nandan/sprint" rel="external noopener nofollow">this https URL</a>.

Representation Sparsification with Hybrid Thresholding for Fast SPLADE-based Document Retrieval

Deep Neural Network Acceleration with Sparse Prediction Layers

Two-Step SPLADE: Simple, Efficient and Effective Approximation of SPLADE

SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking

Approximate Cluster-Based Sparse Document Retrieval with Segmented Maximum Term Weights

Sparse Meets Dense: A Hybrid Approach to Enhance Scientific Document Retrieval

Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval

Optimizing Guided Traversal for Fast Learned Sparse Retrieval

DeeperImpact: Optimizing Sparse Learned Index Structures

Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks

SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval

SpaDE: Improving Sparse Representations using a Dual Document Encoder for First-stage Retrieval

A Static Pruning Study on Sparse Neural Retrievers

Adapting Learned Sparse Retrieval for Long Documents

Sparse, Dense, and Attentional Representations for Text Retrieval

Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understanding

Contextualization with SPLADE for High Recall Retrieval

Topic-Grained Text Representation-based Model for Document Retrieval

SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval

Densifying Sparse Representations for Passage Retrieval by Representational Slicing

Faster Learned Sparse Retrieval with Guided Traversal