Abstract:Sustainability research faces many challenges as respective environmental, urban and regional contexts are experiencing rapid changes at an unprecedented spatial granularity level, which involves growing massive data and the need for spatial relationship detection at a faster pace. Spatial join is a fundamental method for making data more informative with respect to spatial relations. The dramatic growth of data volumes has led to increased focus on high-performance large-scale spatial join. In this paper, we present Spatial Join with Spark (SJS), a proposed high-performance algorithm, that uses a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculation-evaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access. We compare four in-memory spatial join algorithms in SJS for further performance improvement. Based on extensive experiments with real-world data, we conclude that SJS outperforms the Spark and MapReduce implementations of earlier spatial join approaches. This study demonstrates that it is promising to leverage high-performance computing for large-scale spatial join analysis. The availability of large-sized geo-referenced datasets along with the high-performance computing technology can raise great opportunities for sustainability research on whether and how these new trends in data and technology can be utilized to help detect the associated trends and patterns in the human-environment dynamics.

A Novel Methodology for Parallel Spatial Overlay over Vector Data — A Case Study with Shape File

A Scalable Parallel Software Volume Rendering Algorithm for Large-Scale Unstructured Data

Research on Parallel Computing of Spatial Vector Data Conversion Based on Common Interface

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

A Method of Spatial Data Partition for Efficient Parallel Computing of Topological Relations

Parallel Calculation and Efficiency Analysis for Neighborhood Statistic Algorithm in Digital Terrain Analysis

Research Research on on Parallel Algorithm for Polygon Rasterization.

Multi-Core Programming in Land Plan Spatial Data Quality Checking: A Case Study on Multilayer Relationship Checking

A spatial data partition algorithm based on statistical cluster

A Parallel Method to Accelerate Spatial Operations Involving Polygon Intersections.

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Review on Parallel Spatial Analysis Algorithms

A Parallel Algorithm for Erosion Slope Length Calculation Based on Grid Terrain Datasets

Parallel scanline algorithm for rapid rasterization of vector geographic data

Research On The Parallelization Of The Dbscan Clustering Algorithm For Spatial Data Mining Based On The Spark Platform

Parallel Volume Rendering Based on LOD Method

Parallel Algorithm Designed for Polygon Vectorization.

Spatial overlay analysis of land use vector data based on Spark

A novel data decomposition method for rapid parallel processing of vector polygon rasterization

Parallel Visualization for Large-Scale Datasets

Design and Implementation of Rasterization Parallel Algorithm Based on Boundary Algebra Filling