Abstract:Sustainability research faces many challenges as respective environmental, urban and regional contexts are experiencing rapid changes at an unprecedented spatial granularity level, which involves growing massive data and the need for spatial relationship detection at a faster pace. Spatial join is a fundamental method for making data more informative with respect to spatial relations. The dramatic growth of data volumes has led to increased focus on high-performance large-scale spatial join. In this paper, we present Spatial Join with Spark (SJS), a proposed high-performance algorithm, that uses a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculation-evaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access. We compare four in-memory spatial join algorithms in SJS for further performance improvement. Based on extensive experiments with real-world data, we conclude that SJS outperforms the Spark and MapReduce implementations of earlier spatial join approaches. This study demonstrates that it is promising to leverage high-performance computing for large-scale spatial join analysis. The availability of large-sized geo-referenced datasets along with the high-performance computing technology can raise great opportunities for sustainability research on whether and how these new trends in data and technology can be utilized to help detect the associated trends and patterns in the human-environment dynamics.

The Optimization of Remote Spatial Join Queries on Spatial Information Grid

Utility Based Query Dissemination in Spatial Data Grid

New Distributed Spatial Query Optimization Approach by Using Query Analyzer

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

A Distributed Cooperative Technology for Spatial Grid Computing

Parallel Spatial Join Query Processing

An Effective High-Performance Multiway Spatial Join Algorithm with Spark

A SQL transformation based optimization algorithm for distributed spatial query

Vector Spatial Big Data Storage and Optimized Query Based on the Multi-Level Hilbert Grid Index in HBase

Remote Visualization Based On Grid Computing

Study on Replica Catalog Management in Grid Spatial Database

A Study of Performance Optimization Method for Massive Spaito-temporal Data Based on Spatio-temporal Partition Clustering

Agent-Based Spatial Query Optimization in a Grid Environment

Building the Distributed Geographic SQL Workflow in the Grid Environment.

A Grid-Aided and STR-Tree-based Algorithm for Partitioning Vector Data

Optimization Service for Complex Process System in Grid Environment and Its Task Scheduling Strategy

Real-Time Spatial Queries for Moving Objects Using Storm Topology.

Optimizing Grid Computing Configuration and Scheduling for Geospatial Analysis: an Example with Interpolating DEM

Research of the Grid Computing System Applied in Optical Simulation

A Mechanism of Remote Sensing Image for Parallel Processing Base on Splitting Blocks