Abstract:Sustainability research faces many challenges as respective environmental, urban and regional contexts are experiencing rapid changes at an unprecedented spatial granularity level, which involves growing massive data and the need for spatial relationship detection at a faster pace. Spatial join is a fundamental method for making data more informative with respect to spatial relations. The dramatic growth of data volumes has led to increased focus on high-performance large-scale spatial join. In this paper, we present Spatial Join with Spark (SJS), a proposed high-performance algorithm, that uses a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculation-evaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access. We compare four in-memory spatial join algorithms in SJS for further performance improvement. Based on extensive experiments with real-world data, we conclude that SJS outperforms the Spark and MapReduce implementations of earlier spatial join approaches. This study demonstrates that it is promising to leverage high-performance computing for large-scale spatial join analysis. The availability of large-sized geo-referenced datasets along with the high-performance computing technology can raise great opportunities for sustainability research on whether and how these new trends in data and technology can be utilized to help detect the associated trends and patterns in the human-environment dynamics.

Algorithms to Visualizing Large Spatial Data Set

Using Visualization to Improve Clustering Analysis on Heterogeneous Information Network.

Parallel Visualization for Large-Scale Datasets

Visualizing Large-Scale and High-Dimensional Data

The Visibility Techniques for Large-Scale Scene

Efficient Visualization Techniques for High Resolution Remotely Sensed Data in A Network Environment

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

HiVision: Rapid visualization of large-scale spatial vector data

An Algorithm for Real-Time Visualization of Large-Scale Terrain

Study of GIS-based Calculation of Flood Area and Visualization of Virtual Reality

Visual Clustering in Parallel Coordinates

A Quick and Feature based Visualization Algorithm for Large-scale Flow Data.

A method of the dense-sparse vector tile generation accounting for the spatial distribution of features

A Study of Real Time Raster Graphic Visualization Based on Big Data Technology

Analysis of Statistics Data Based on Mixed Visualization Techniques

A three-dimensional display for big data sets

Visualizing Large-Scale High-Dimensional Data Via Hierarchical Embedding of KNN Graphs

A spatial data partition algorithm based on statistical cluster

3D Visualization of Massive Terrain Data Based on Grid Partition

Scale Space Based on Clustering Method Integrating Spatial Relationships and Non-Spatial Atrributes

An effective visualization method for large-scale terrain dataset