Abstract:Spatial data partitioning (SDP) plays a powerful role in distributed storage and parallel computing for spatial data. However, due to skew distribution of spatial data and varying volume of spatial vector objects, it leads to a significant challenge to ensure both optimal performance of spatial operation and data balance in the cluster. To tackle this problem, we proposed a spatial coding-based approach for partitioning big spatial data in Hadoop. This approach, firstly, compressed the whole big spatial data based on spatial coding matrix to create a sensing information set (SIS), including spatial code, size, count and other information. SIS was then employed to build spatial partitioning matrix, which was used to spilt all spatial objects into different partitions in the cluster finally. Based on our approach, the neighbouring spatial objects can be partitioned into the same block. At the same time, it also can minimize the data skew in Hadoop distributed file system (HDFS). The presented approach with a case study in this paper is compared against random sampling based partitioning, with three measurement standards, namely, the spatial index quality, data skew in HDFS, and range query performance. The experimental results show that our method based on spatial coding technique can improve the query performance of big spatial data, as well as the data balance in HDFS. We implemented and deployed this approach in Hadoop, and it is also able to support efficiently any other distributed big spatial data systems.

A Packaging Approach for Massive Amounts of Small Geospatial Files with HDFS.

The Framework of a Distributed File System for Geospatial Data Management.

Efficient Spatial Big Data Storage and Query in HBase.

An Optimized Approach for Storing and Accessing Small Files on Cloud Storage

SQL-DFS: A Massive Small File Storage System Based on HDFS

HBaseSpatial: A Scalable Spatial Data Storage Based on HBase

Sfs: A Massive Small File Processing Middleware in Hadoop

A Strategy to Deal with Mass Small Files in HDFS

A Strategy for Small Files Processing in HDFS.

A Method to Improve the Performance for Storing Massive Small Files in Hadoop

Storage Optimization Method of Small Files Based on Hadoop

Mosm: An Approach For Efficient Storing Massive Small Files On Hadoop

An Optimized Method of HDFS for Massive Small Files Storage

Efficient File Accessing Techniques on Hadoop Distributed File Systems.

Hadoop Massive Small File Merging Technology Based on Visiting Hot-Spot and Associated File Optimization

LHF: A New Archive Based Approach to Accelerate Massive Small Files Access Performance in HDFS.

Spatial Coding-Based Approach for Partitioning Big Spatial Data in Hadoop

A Study of Performance Optimization Method for Massive Spaito-temporal Data Based on Spatio-temporal Partition Clustering

Research on the Analysis and Statistic of Geographical Conditions Based on the Strategy of "grid Index + MapReduce"

Study of Seamless Organization and Storage Structure for Massive Spatio-Temporal Data

An Optimized Approach for Storing Small Files on HDFS-based on Dynamic Queue