Abstract:With increasing numbers of GPS-equipped mobile devices, we are witnessing a deluge of spatial information that needs to be effectively and efficiently managed. Even though there are several distributed spatial data processing systems such as GeoSpark (Apache Sedona), the effects of underlying storage engines have not been well studied for spatial data processing. In this paper, we evaluate the performance of various distributed storage engines for processing large-scale spatial data using GeoSpark, a state-of-the-art distributed spatial data processing system running on top of Apache Spark. For our performance evaluation, we choose three distributed storage engines having different characteristics: (1) HDFS, (2) MongoDB, and (3) Amazon S3. To conduct our experimental study on a real cloud computing environment, we utilize Amazon EMR instances (up to 6 instances) for distributed spatial data processing. For the evaluation of big spatial data processing, we generate data sets considering four kinds of various data distributions and various data sizes up to one billion point records (38.5GB raw size). Through the extensive experiments, we measure the processing time of storage engines with the following variations: (1) sharding strategies in MongoDB, (2) caching effects, (3) data distributions, (4) data set sizes, (5) the number of running executors and storage nodes, and (6) the selectivity of queries. The major points observed from the experiments are summarized as follows. (1) The overall performance of MongoDB-based GeoSpark is degraded compared to HDFS- and S3-based GeoSpark in our experimental settings. (2) The performance of MongoDB-based GeoSpark is relatively improved in large-scale data sets compared to the others. (3) HDFS- and S3-based GeoSpark are more scalable to running executors and storage nodes compared to MongoDB-based GeoSpark. (4) The sharding strategy based on the spatial proximity significantly improves the performance of MongoDB-based GeoSpark. (5) S3- and HDFS-based GeoSpark show similar performances in all the environmental settings. (6) Caching in distributed environments improves the overall performance of spatial data processing. These results can be usefully utilized in decision-making of choosing the most adequate storage engine for big spatial data processing in a target distributed environment.

Beyond Click-and-View: a Comparative Study of Data Management Approaches for Interactive Visualization

FlashView

Suitability of Graph Database Technology for the Analysis of Spatio-Temporal Data

Supporting Interactive Analytics and Visualization on Large Data

Combining Visual Analytics and Content Based Data Retrieval Technology for Efficient Data Analysis

A comparison of data visualization tools: A case study in health-related research

A comparative experimental study of distributed storage engines for big spatial data processing using GeoSpark

A New Design of High-Performance Large-Scale GIS Computing at a Finer Spatial Granularity: A Case Study of Spatial Join with Spark for Sustainability

A Visual Analytics Approach to Understanding Spatiotemporal Hotspots

Research on Temporal Query Expansion and Temporal Index Optimization Based on Spark

GeoSpark SQL: an Effective Framework Enabling Spatial Queries on Spark

A Comparison of Spatiotemporal Visualizations for 3D Urban Analytics

A high performance web-based system for analyzing and visualizing spatiotemporal data for climate studies

OM3: An Ordered Multi-level Min-Max Representation for Interactive Progressive Visualization of Time Series.

Viewpoints: A high-performance high-dimensional exploratory data analysis tool

Visualization And Diagnosis Of Earth Science Data Through Hadoop And Spark

Evaluating NoSQL Databases for OLAP Workloads: A Benchmarking Study of MongoDB, Redis, Kudu and ArangoDB

MAP-Vis: A Distributed Spatio-Temporal Big Data Visualization Framework Based on a Multi-Dimensional Aggregation Pyramid Model

Optimizing Dataflow Systems for Scalable Interactive Visualization

Large vector spatial data storage and query processing using clickhouse

A Framework to Visualize Temporal Behavioral Relationships in Streaming Multivariate Data