Abstract:The kNN join problem, denoted by R x(KNN) S, is to find the k nearest neighbors from a given dataset S for each point in the query set R. It is an operation required by many big data applications. As large volume of data are continuously generated in more and more real-life cases, we address the problem of monitoring kNN join results on data streams. Specifically, we are concerned with answering kNN join periodically at each snapshot which is called snapshot kNN join. Existing kNN join solutions mainly solve the problem on static datasets, or on a single centralized machine, which are difficult to scale to large data on data streams. In this paper, we propose to incrementally calculate the kNN join results of time t(i) from the results of snapshot t(i-1). Typically, for the data continuously generated on the data stream, we can get S-i = Si-1 + Delta S-i for the valid datasets of adjacent snapshots, where Delta S-i denotes the updated points between time t(i -1) and t(i). Our basic idea is to first find the queries in R whose kNN results can be affected by the updated points in Delta S-i, and then update the kNN results of these small part of queries respectively. In this way, we can avoid calculating the kNN join results on the whole dataset S-i in time t(i). We propose an implementation of searching for affected query points in MapReduce to scale to large volume of data. In brief, the mappers partition the datasets into groups, and the reducers search for affected queries separately on each group of points. Furthermore, we present the enhanced strategies of data partitioning and grouping to reduce the shuffling cost and computational cost. Extensive experiments on real-world datasets demonstrate that our proposed methods are efficient, robust, and scalable.

Efficient index-based KNN join processing for high-dimensional data

Efficient K-Nearest Neighbor Join Algorithms for High Dimensional Sparse Data

High-dimensional kNN joins with incremental updates

Exploring Bit-Difference for Approximate KNN Search in High-Dimensional Databases

BC-iDistance： an optimized high-dimensional index for KNN processing

Composite Distance Transformation for Indexing and K -Nearest-neighbor Searching in High-Dimensional Spaces

iDistance: An adaptive B+-tree based indexing method for nearest neighbor search

Preserving-Ignoring Transformation Based Index for Approximate k Nearest Neighbor Search

K Nearest Neighbor Queries and Knn-Joins in Large Relational Databases (almost) for Free

Efficient Processing of k Nearest Neighbor Joins using MapReduce

Efficient Parallel Processing of High-Dimensional Spatial K NN Queries

Enhancing K-nearest neighbor algorithm: a comprehensive review and performance analysis of modifications

Efficient Data-aware Distance Comparison Operations for High-Dimensional Approximate Nearest Neighbor Search

Diagonal Ordering: A New Approach to High-Dimensional KNN Processing.

Adaptive Quantization Of The High-Dimensional Data For Efficient Knn Processing

An Improved KNN Algorithm Based on Adaptive Cluster Distance Bounding for High Dimensional Indexing

Effective optimizations of cluster-based nearest neighbor search in high-dimensional space

BC-iDistance:Bit-code Based Optimal High-dimensional Index

Efficient Nearest Neighbor Query Based on Extended B-tree in High-Dimensional Space.

Contorting High Dimensional Data for Efficient Main Memory KNN Processing

Efficient Snapshot Knn Join Processing for Large Data Using Mapreduce