Abstract:Vector retrieval focuses on finding the k-nearest neighbors from a bunch of data points, and is widely used in a diverse set of areas such as information retrieval and recommender system. The current state-of-the-art methods represented by HNSW usually generate indexes with a big memory footprint, restricting the scale of data they can handle, except resorting to a hybrid index with external storage. The space-partitioning learned indexes, which only occupy a small memory, have made great breakthroughs in recent years. However, these methods rely on a large amount of labeled data for supervised learning, so model complexity affects the generalization. To this end, we propose a lightweight learnable hierarchical space partitioning index based on a balanced K-ary tree, called BAlanced Tree Learner (BATL), where the same bucket of data points are represented by a path from the root to the corresponding leaf. Instead of mapping each query into a bucket, BATL classifies it into a sequence of branches (i.e. a path), which drastically reduces the number of classes and potentially improves generalization. BATL updates the classifier and the balanced tree in an alternating way. When updating the classifier, we innovatively leverage the sequence-to-sequence learning paradigm for learning to route each query into the ground-truth leaf on the balanced tree. Retrieval is then boiled down into a sequence (i.e. path) generation task, which can be simply achieved by beam search on the encoder-decoder. When updating a balanced tree, we apply the classifier for navigating each data point into the tree nodes layer by layer under the balance constraints. We finally evaluate BATL with several large-scale vector datasets, where the experimental results show the superiority of the proposed method to the SOTA baselines in the tradeoff among latency, accuracy, and memory cost.

Searching Dense Representations with Inverted Indexes

ESPN: Memory-Efficient Multi-Vector Information Retrieval

Scalable Top-K Spatial Keyword Search

Vector Search with OpenAI Embeddings: Lucene Is All You Need

Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations

Operational Advice for Dense and Sparse Retrievers: HNSW, Flat, or Inverted Indexes?

Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighbors

The Impacts of Data, Ordering, and Intrinsic Dimensionality on Recall in Hierarchical Navigable Small Worlds

EHI: End-to-end Learning of Hierarchical Index for Efficient Dense Retrieval

Pairing Clustered Inverted Indexes with kNN Graphs for Fast Approximate Retrieval over Learned Sparse Representations

Multiple Complementary Inverted Indexing Based on Multiple Metrics

Efficient Neural Ranking using Forward Indexes and Lightweight Encoders

Indexing of the CNN Features for the Large Scale Image Search

Learning Passage Impacts for Inverted Indexes

Efficient Reverse $k$ Approximate Nearest Neighbor Search over High-Dimensional Vectors

Processing Long Queries Against Short Text

Dense Sparse Retrieval: Using Sparse Language Models for Inference Efficient Dense Retrieval

Learning to Search Efficiently in High Dimensions.

On Single and Multiple Representations in Dense Passage Retrieval

Learning Balanced Tree Indexes for Large-Scale Vector Retrieval

Hierarchical indexing scheme for fast search in a large-scale image database