Abstract:The Big Data explosion has necessitated the development of search algorithms that scale sub-linearly in time and memory. While compression algorithms and search algorithms do exist independently, few algorithms offer both, and those which do are domain-specific. We present panCAKES, a novel approach to compressive search, i.e., a way to perform $k$-NN and $\rho$-NN search on compressed data while only decompressing a small, relevant, portion of the data. panCAKES assumes the manifold hypothesis and leverages the low-dimensional structure of the data to compress and search it efficiently. panCAKES is generic over any distance function for which the distance between two points is proportional to the memory cost of storing an encoding of one in terms of the other. This property holds for many widely-used distance functions, e.g. string edit distances (Levenshtein, Needleman-Wunsch, etc.) and set dissimilarity measures (Jaccard, Dice, etc.). We benchmark panCAKES on a variety of datasets, including genomic, proteomic, and set data. We compare compression ratios to gzip, and search performance between the compressed and uncompressed versions of the same dataset. panCAKES achieves compression ratios close to those of gzip, while offering sub-linear time performance for $k$-NN and $\rho$-NN search. We conclude that panCAKES is an efficient, general-purpose algorithm for exact compressive search on large datasets that obey the manifold hypothesis. We provide an open-source implementation of panCAKES in the Rust programming language.

Fast KNN search for big data with set compression tree and best bin first

Constrained All-k-Nearest-Neighbor Search

Two-point-based binary search trees for accelerating big data classification using KNN

Adaptive Quantization Of The High-Dimensional Data For Efficient Knn Processing

KPB-SIFT

Generalized compression and compressive search of large datasets

Contorting High Dimensional Data for Efficient Main Memory KNN Processing

Composite Distance Transformation for Indexing and K -Nearest-neighbor Searching in High-Dimensional Spaces

Fast Nearest Neighbor Search in the Hamming Space.

Hierarchical indexing scheme for fast search in a large-scale image database

Optimizing Kd-Trees for Scalable Visual Descriptor Indexing

Hierarchical Indexing Scheme for Fast Search in Large-Scale Image Database

Fast answering k-nearest-neighbor queries over large image databases using dual distance transformation

Large Scale Nearest Neighbors Search Based on Neighborhood Graph.

Fast approximate K neighbor method based on tree strategy and balanced K-means clustering

Fast Content-Based Image Retrieval Based on Equal-Average K-Nearest-Neighbor Search Schemes

An Encoding-Based Dual Distance Tree High-Dimensional Index

Exploring Bit-Difference for Approximate KNN Search in High-Dimensional Databases

Indexing high-dimensional data in dual distance spaces: a symmetrical encoding approach

Norm-Based Binary Search Trees for Speeding Up KNN Big Data Classification

$\Sf {simknn}$: A Scalable Method for In-Memoryknn Search over Moving Objects in Road Networks.