Abstract:Sustainability research faces many challenges as respective environmental, urban and regional contexts are experiencing rapid changes at an unprecedented spatial granularity level, which involves growing massive data and the need for spatial relationship detection at a faster pace. Spatial join is a fundamental method for making data more informative with respect to spatial relations. The dramatic growth of data volumes has led to increased focus on high-performance large-scale spatial join. In this paper, we present Spatial Join with Spark (SJS), a proposed high-performance algorithm, that uses a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculation-evaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access. We compare four in-memory spatial join algorithms in SJS for further performance improvement. Based on extensive experiments with real-world data, we conclude that SJS outperforms the Spark and MapReduce implementations of earlier spatial join approaches. This study demonstrates that it is promising to leverage high-performance computing for large-scale spatial join analysis. The availability of large-sized geo-referenced datasets along with the high-performance computing technology can raise great opportunities for sustainability research on whether and how these new trends in data and technology can be utilized to help detect the associated trends and patterns in the human-environment dynamics.

An adaptive skew insensitive join algorithm for large scale data analytics

An Adaptive Skew Handling Join Algorithm for Large-scale Data Analysis

Join Query Optimization Based on MapReduce under Skewed Data

Optimization Factor Analysis Of Large-Scale Join Queries On Different Platforms

A Request Skew Aware Heterogeneous Distributed Storage System Based on Cassandra

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

Efficient Join Synopsis Maintenance for Data Warehouse.

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Skew Strikes Back: New Developments in the Theory of Join Algorithms

A cost aware adaptive multiple table join evaluation in MapReduce

A Real-Time Partition Generation Mechanism for Data Skew Mitigation in Spark Computing Environment

Skyline-Join in Distributed Databases

Improving Distributed Similarity Join in Metric Space with Error-bounded Sampling

JoinSketch: A Sketch Algorithm for Accurate and Unbiased Inner-Product Estimation.

SkewControl: Gini out of the Bottle.

Joinable Search over Multi-source Spatial Datasets: Overlap, Coverage, and Efficiency

Utilizing the column imprints to accelerate no‐partitioning hash joins in large‐scale edge systems

Efficient Join Processing Over Incomplete Data Streams (Technical Report)

A Dimension Reduction Technique for Large-scale Structured Sparse Optimization Problems with Application to Convex Clustering

Instance-Optimal Acyclic Join Processing Without Regret: Engineering the Yannakakis Algorithm in Column Stores

A learning-based framework for spatial join processing: estimation, optimization and tuning