Abstract:Recently, MapReduce based spatial query systems have emerged as a cost effective and scalable solution to large scale spatial data processing and analytics. MapReduce based systems achieve massive scalability by partitioning the data and running query tasks on those partitions in parallel. Therefore, effective data partitioning is critical for task parallelization, load balancing, and directly affects system performance. However, several pitfalls of spatial data partitioning make this task particularly challenging. First, data skew is very common in spatial applications. To achieve best query performance, data skew need to be reduced. Second, spatial partitioning approaches generate boundary objects that cross multiple partitions, and add extra query processing overhead. Consequently, boundary objects need to be minimized. Third, the high computational complexity of spatial partitioning algorithms combined with massive amounts of data require an efficient approach for partitioning to achieve overall fast query response. In this paper, we provide a systematic evaluation of multiple spatial partitioning methods with a set of different partitioning strategies, and study their implications on the performance of MapReduce based spatial queries. We also study sampling based partitioning methods and their impact on queries, and propose several MapReduce based high performance spatial partitioning methods. The main objective of our work is to provide a comprehensive guidance for optimal spatial data partitioning to support scalable and fast spatial data processing in massively parallel data processing frameworks such as MapReduce. The algorithms developed in this work are open source and can be easily integrated into different high performance spatial data processing systems.

Accelerating Spatial Data Processing with MapReduce

Effective Spatial Data Partitioning for Scalable Query Processing

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Efficient Multi-dimensional Spatial RkNN Query Processing with MapReduce

Elastic and Effective Spatio-Temporal Query Processing Scheme on Hadoop

Research on the Analysis and Statistic of Geographical Conditions Based on the Strategy of "grid Index + MapReduce"

A MapReduce approach to G i *( d ) spatial statistic

An efficient PAM spatial clustering algorithm based on MapReduce

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Inverted Voronoi-Based Knn Query Processing With Mapreduce

The Performance of MapReduce

ELM-MapReduce: MapReduce Accelerated Extreme Learning Machine for Big Spatial Data Analysis

The performance of MapReduce: an in-depth study

Spatial-Htm: A Mapreduce-Based System For Querying Spatial Data With The Hierarchical Triangular Mesh

A Parallel Implementation of the K-Means Algorithm Based on MapReduce

An Efficient Top-k Spatial Join Query Processing Algorithm on Big Spatial Data.

MapReduce for astronomical cross-matching

Towards Parallel Spatial Query Processing for Big Spatial Data.

SDPPF — A MapReduce based parallel processing framework for spatial data

Beyond Simple Integration of RDBMS and MapReduce -- Paving the Way toward a Unified System for Big Data Analytics: Vision and Progress

Irregular Partitioning Method Based K-Nearest Neighbor Query Algorithm Using Mapreduce