Abstract:In recent years, Approximate Nearest Neighbor Search (ANNS) has played a pivotal role in modern search and recommendation systems, especially in emerging LLM applications like Retrieval-Augmented Generation. There is a growing exploration into harnessing the parallel computing capabilities of GPUs to meet the substantial demands of ANNS. However, existing systems primarily focus on offline scenarios, overlooking the distinct requirements of online applications that necessitate real-time insertion of new vectors. This limitation renders such systems inefficient for real-world scenarios. Moreover, previous architectures struggled to effectively support real-time insertion due to their reliance on serial execution streams. In this paper, we introduce a novel Real-Time Adaptive Multi-Stream GPU ANNS System (RTAMS-GANNS). Our architecture achieves its objectives through three key advancements: 1) We initially examined the real-time insertion mechanisms in existing GPU ANNS systems and discovered their reliance on repetitive copying and memory allocation, which significantly hinders real-time effectiveness on GPUs. As a solution, we introduce a dynamic vector insertion algorithm based on memory blocks, which includes in-place rearrangement. 2) To enable real-time vector insertion in parallel, we introduce a multi-stream parallel execution mode, which differs from existing systems that operate serially within a single stream. Our system utilizes a dynamic resource pool, allowing multiple streams to execute concurrently without additional execution blocking. 3) Through extensive experiments and comparisons, our approach effectively handles varying QPS levels across different datasets, reducing latency by up to 40%-80%. The proposed system has also been deployed in real-world industrial search and recommendation systems, serving hundreds of millions of users daily, and has achieved good results.

GPU-utilized Composite Quantization for Large-scale High-dimension ANN Search

Large-scale Image Retrieval Based on Boosting Iterative Quantization Hashing with Query-Adaptive Reranking.

Hierarchical quantization for billion-scale similarity retrieval on GPUs

Efficient Large-scale Approximate Nearest Neighbor Search on the GPU

Vector and Line Quantization for Billion-scale Similarity Search on GPUs

JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mapping

GPU-accelerated Proximity Graph Approximate Nearest Neighbor Search and Construction

FusionANNS: An Efficient CPU/GPU Cooperative Processing Architecture for Billion-scale Approximate Nearest Neighbor Search

Efficient Large-Scale Approximate Nearest Neighbor Search on OpenCL FPGA

Accelerating Massive Queries of Approximate Nearest Neighbor Search on High-Dimensional Data

Hash Learning with Variable Quantization for Large-scale Retrieval

DiskANN++: Efficient Page-based Search over Isomorphic Mapped Graph Index using Query-sensitivity Entry Vertex

BANG: Billion-Scale Approximate Nearest Neighbor Search using a Single GPU

SymphonyQG: Towards Symphonious Integration of Quantization and Graph for Approximate Nearest Neighbor Search

Artificial Neural Network Computation On Graphic Process Unit

Accelerated Nearest Neighbor Search with Quick ADC

Projected Residual Vector Quantization for ANN Search

Learned Probing Cardinality Estimation for High-Dimensional Approximate NN Search.

A Real-Time Adaptive Multi-Stream GPU System for Online Approximate Nearest Neighborhood Search

Improving Bilayer Product Quantization for Billion-Scale Approximate Nearest Neighbors in High Dimensions

Sparse Composite Quantization