Abstract:Sustainability research faces many challenges as respective environmental, urban and regional contexts are experiencing rapid changes at an unprecedented spatial granularity level, which involves growing massive data and the need for spatial relationship detection at a faster pace. Spatial join is a fundamental method for making data more informative with respect to spatial relations. The dramatic growth of data volumes has led to increased focus on high-performance large-scale spatial join. In this paper, we present Spatial Join with Spark (SJS), a proposed high-performance algorithm, that uses a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculation-evaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access. We compare four in-memory spatial join algorithms in SJS for further performance improvement. Based on extensive experiments with real-world data, we conclude that SJS outperforms the Spark and MapReduce implementations of earlier spatial join approaches. This study demonstrates that it is promising to leverage high-performance computing for large-scale spatial join analysis. The availability of large-sized geo-referenced datasets along with the high-performance computing technology can raise great opportunities for sustainability research on whether and how these new trends in data and technology can be utilized to help detect the associated trends and patterns in the human-environment dynamics.

An Efficient Splicing and Rapid Extraction of Spatial Data Method Based on Global Subdivsion

Rapid Collection Method of Multi-Source Data Based on Global Subdivision Grid

A method of spatial association for multi-sources remote sensing data based on global subdivision grid

Research of Applying GeoSOT Global Subdivision Grids Method on Fast Mosaic of Remote Sensing Data in Large Area

Spatial Data Management Method with GeoSOT Grid

An Index and Retrieval Method of Spatial Data Based on GeoSOT Global Discrete Grid System

A Study of Performance Optimization Method for Massive Spaito-temporal Data Based on Spatio-temporal Partition Clustering

A Full-text Retrieval Method for Spatial Data Search Based on Global Subdivision Grid

Study of Seamless Organization and Storage Structure for Massive Spatio-Temporal Data

Research on the Analysis and Statistic of Geographical Conditions Based on the Strategy of "grid Index + MapReduce"

Spatial Data Coding Method Based on Global Subdivision Grid

Method for Managing and Querying Geo-Spatial Data Using a Grid-Code-array Spatial Index

A Study on Data Storage and Management for Massive Remote Sensing Data Based on Multi-level Grid Model

Integration and Management of Massive Remote-Sensing Data Based on GeoSOT Subdivision Model

A grid-aided and STR-Tree-based algorithm for partitioning vector data

Area-location Association and Comprehensive Expression of Multi-source Spatial Information Based on Global Subdivision Framework

New Method Adaptive to Geospatial Information Acquisition and Share Based on Grid

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

The Three-Dimensional Data Organization Method Based on GeoSOT-3D

A unified accessing method of multi-resolution grid point data for tiled map services