Abstract:Sustainability research faces many challenges as respective environmental, urban and regional contexts are experiencing rapid changes at an unprecedented spatial granularity level, which involves growing massive data and the need for spatial relationship detection at a faster pace. Spatial join is a fundamental method for making data more informative with respect to spatial relations. The dramatic growth of data volumes has led to increased focus on high-performance large-scale spatial join. In this paper, we present Spatial Join with Spark (SJS), a proposed high-performance algorithm, that uses a simple, but efficient, uniform spatial grid to partition datasets and joins the partitions with the built-in join transformation of Spark. SJS utilizes the distributed in-memory iterative computation of Spark, then introduces a calculation-evaluating model and in-memory spatial repartition technology, which optimize the initial partition by evaluating the calculation amount of local join algorithms without any disk access. We compare four in-memory spatial join algorithms in SJS for further performance improvement. Based on extensive experiments with real-world data, we conclude that SJS outperforms the Spark and MapReduce implementations of earlier spatial join approaches. This study demonstrates that it is promising to leverage high-performance computing for large-scale spatial join analysis. The availability of large-sized geo-referenced datasets along with the high-performance computing technology can raise great opportunities for sustainability research on whether and how these new trends in data and technology can be utilized to help detect the associated trends and patterns in the human-environment dynamics.

Large-Scale Social Network Analysis Based on MapReduce

Distributed Affinity Propagation Clustering Based on MapReduce

Research on method for extracting large-scale social network based on Mapreduce

Evaluating Large Graph Processing in MapReduce Based on Message Passing

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Community structure mining in big data social media networks with MapReduce

Distributed Centrality Analysis of Social Network Data Using MapReduce

A Parallel Computing Model for Large-Graph Mining with MapReduce.

LI-MR: A Local Iteration Map/Reduce Model and Its Application to Mine Community Structure in Large-Scale Networks

Network Public Opinion Monitoring Based on MapReduce

X-RIME: Cloud-Based Large Scale Social Network Analysis

KANGAROO: A Distributed System for SNA - Social Network Analysis in Huge-scale Networks

Unstructured Data Analysis on Big Data Using Map Reduce

Distributed structural clustering on large graph

Exploratory Analysis for Big Social Data Using Deep Network

Distributed data management using MapReduce

Scalable Community Discovery of Large Networks

Large-scale Data Mining Method based on Clustering Algorithm Combined with MAPREDUCE

A Two-Tier Partition Algorithm for the Optimization of the Large-Scale Simulation of Information Diffusion in Social Networks.

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration