Abstract:Billion-scale high-dimensional approximate nearest neighbour (ANN) search has become an important problem for searching similar objects among the vast amount of images and videos available online. The existing ANN methods are usually characterized by their specific indexing structures, including the inverted index and the inverted multi-index structure. The inverted index structure is amenable to GPU-based implementations, and the state-of-the-art systems such as Faiss are able to exploit the massive parallelism offered by GPUs. However, the inverted index requires high memory overhead to index the dataset effectively. The inverted multi-index structure is difficult to implement for GPUs, and also ineffective in dealing with database with different data distributions. In this paper we propose a novel hierarchical inverted index structure generated by vector and line quantization methods. Our quantization method improves both search efficiency and accuracy, while maintaining comparable memory consumption. This is achieved by reducing search space and increasing the number of indexed regions. We introduce a new ANN search system, VLQ-ADC, that is based on the proposed inverted index, and perform extensive evaluation on two public billion-scale benchmark datasets SIFT1B and DEEP1B. Our evaluation shows that VLQ-ADC significantly outperforms the state-of-the-art GPU- and CPU-based systems in terms of both accuracy and search speed. The source code of VLQ-ADC is available at <a class="link-external link-https" href="https://github.com/zjuchenwei/vector-line-quantization" rel="external noopener nofollow">this https URL</a>.

Aggregating Tree for Searching in Billion Scale High Dimensional Data

Accelerating Exact Nearest Neighbor Search in High Dimensional Euclidean Space Via Block Vectors

Hierarchical quantization for billion-scale similarity retrieval on GPUs

An Encoding-Based Dual Distance Tree High-Dimensional Index

Fast High-dimensional Approximate Nearest Neighbor Search with Efficient Index Time and Space

A Compact Binary Aggregated Descriptor Via Dual Selection for Visual Search

Distributed Adaptive Binary Quantization for Fast Nearest Neighbor Search.

Indexing High-Dimensional Data in Dual Distance Spaces

Indexing high-dimensional data in dual distance spaces: a symmetrical encoding approach

Hash Learning with Variable Quantization for Large-scale Retrieval

Adaptive Quantization Of The High-Dimensional Data For Efficient Knn Processing

Double-Bit Quantization and Index Hashing for Nearest Neighbor Search

Hierarchical indexing scheme for fast search in a large-scale image database

Indexing very high-dimensional sparse and quasi-sparse vectors for similarity searches

Hierarchical Indexing Scheme for Fast Search in Large-Scale Image Database

Trinary-projection trees for approximate nearest neighbor search.

Composite Distance Transformation for Indexing and K -Nearest-neighbor Searching in High-Dimensional Spaces

Tree-based compact hashing for approximate nearest neighbor search

Vector and Line Quantization for Billion-scale Similarity Search on GPUs

GPU-utilized Composite Quantization for Large-scale High-dimension ANN Search

Affinity Preserving Quantization For Hashing: A Vector Quantization Approach To Learning Compact Binary Codes