Abstract:Recently, MapReduce based spatial query systems have emerged as a cost effective and scalable solution to large scale spatial data processing and analytics. MapReduce based systems achieve massive scalability by partitioning the data and running query tasks on those partitions in parallel. Therefore, effective data partitioning is critical for task parallelization, load balancing, and directly affects system performance. However, several pitfalls of spatial data partitioning make this task particularly challenging. First, data skew is very common in spatial applications. To achieve best query performance, data skew need to be reduced. Second, spatial partitioning approaches generate boundary objects that cross multiple partitions, and add extra query processing overhead. Consequently, boundary objects need to be minimized. Third, the high computational complexity of spatial partitioning algorithms combined with massive amounts of data require an efficient approach for partitioning to achieve overall fast query response. In this paper, we provide a systematic evaluation of multiple spatial partitioning methods with a set of different partitioning strategies, and study their implications on the performance of MapReduce based spatial queries. We also study sampling based partitioning methods and their impact on queries, and propose several MapReduce based high performance spatial partitioning methods. The main objective of our work is to provide a comprehensive guidance for optimal spatial data partitioning to support scalable and fast spatial data processing in massively parallel data processing frameworks such as MapReduce. The algorithms developed in this work are open source and can be easily integrated into different high performance spatial data processing systems.

Irregular Partitioning Method Based K-Nearest Neighbor Query Algorithm Using Mapreduce

Distributed Affinity Propagation Clustering Based on MapReduce

Join Query Optimization Based on MapReduce under Skewed Data

A Parallel Varied Density-Based Clustering Algorithm with Optimized Data Partition

Efficient Processing of k Nearest Neighbor Joins using MapReduce

Discovery of Regional Co-location Patterns with k-Nearest Neighbor Graph.

Design And Implementation Of A Mobile-Health Call System Based On Scalable Knn Query

An Improved K-means Algorithm Based on Mapreduce and Grid

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

An Efficient K-Means Clustering Algorithm On Mapreduce

An efficient PAM spatial clustering algorithm based on MapReduce

Effective Spatial Data Partitioning for Scalable Query Processing

The performance of MapReduce: an in-depth study

The Performance of MapReduce

Efficient Parallel Processing of High-Dimensional Spatial K NN Queries

A spatial data partition algorithm based on statistical cluster

Query optimization for massively parallel data processing.

MPR — A Partitioning-Replication Framework for Multi-Processing kNN Search on Road Networks

DIM: a Distributed Air Index Based on MapReduce for Spatial Query Processing in Road Networks.

Fast Clustering using MapReduce

Large-Scale Social Network Analysis Based on MapReduce