Abstract:ABSTRACTLarge web search engines are facing formidable performance challenges because they have to process thousands of queries per second on tens of billions of documents, within interactive response time. Among many others, Top-k query processing (also called early termination or dynamic pruning) is an important class of optimization techniques that can improve the search efficiency and achieve faster query processing by avoiding the scoring of documents that are unlikely to be in the top results. One recent technique is using Block-Max index. In the Block-Max index, the posting lists are organized as blocks and the maximum score for each block is stored to improve the query efficiency. Although query processing speedup is achieved with Block-Max index, the ranking function for the Top-k results is the term-based approach. It is well known that documents' static scores are also important for a good ranking function. In this paper, we show that the performance of the state-of-the-art algorithms with the Block-Max index is degraded when the static score is added in the ranking function. Then we study efficient techniques for Top-k query processing in the case where a page's static score is given, such as PageRank, in addition to the term-based approach. In particular, we propose a set of new algorithms based on the WAND and MaxScore with Block-Max index using local score, which outperform the existing ones. Then we propose new techniques to estimate a better score upper bound for each block. We also study the search efficiency on different index structures where the document identifiers are assigned by URL sorting or by static document scores. Experiments on TREC GOV2 and ClueWeb09B show that considerable performance gains are achieved.

Faster Learned Sparse Retrieval with Block-Max Pruning

A Static Pruning Study on Sparse Neural Retrievers

Optimizing Guided Traversal for Fast Learned Sparse Retrieval

Faster Learned Sparse Retrieval with Guided Traversal

Dense Sparse Retrieval: Using Sparse Language Models for Inference Efficient Dense Retrieval

Two-Step SPLADE: Simple, Efficient and Effective Approximation of SPLADE

Processing Long Queries Against Short Text

DeeperImpact: Optimizing Sparse Learned Index Structures

Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations

Static Pruning in Dense Retrieval using Matrix Decomposition

A Unified Framework for Learned Sparse Retrieval

Approximate Cluster-Based Sparse Document Retrieval with Segmented Maximum Term Weights

Optimized top-k processing with global page scores on block-max indexes.

SLBRIN: A Spatial Learned Index Based on BRIN

Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently

Towards Order-Preserving Submatrix Search And Indexing

An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models

ILCR: Item-based Latent Factors for Sparse Collaborative Retrieval

Fast Image Retrieval: Query Pruning and Early Termination

Progressive Query Expansion for Retrieval Over Cost-constrained Data Sources