Abstract:Historical earth observation (EO) data have played an important role in long-term scientific and environmental monitoring. The effective organization of large-scale and long-term remote-sensing data to achieve efficient retrieval and access has become one of the important issues. However, inherent big data characteristics, such as a large scale, and asymmetrical temporal and spatial distributions, have caused problems with the efficiency of data retrieval and access. Therefore, this study proposes an efficient data organization method for use in a cloud-computing environment that has two aims. First, it addresses the problem of low retrieval efficiency. An asymmetrical index model for the image metadata is constructed that is based on a unified spatio-temporal grid coding; a prepartitioning mechanism under the HBase architecture is established to realize the uniform storage of the metadata with an asymmetrical spatiotemporal distribution and to avoid retrieval efficiency bottlenecks caused by a load imbalance. Second, it addresses low access efficiency. By dividing the remote-sensing image into tiles, a unified spatio-temporal code is established for each tile, and a consistent hash operation is performed; tiles with similar hash values are stored in the same or adjacent Hadoop Distributed File System nodes. In this way, tiles with temporal or spatial correlations can be gathered and stored, and lots of disk seeks can be avoided during retrieval, thereby greatly improving the data access efficiency. Comparative experiments showed that the data organization method can effectively improve the retrieval and access efficiencies of large-scale and long time-series remote-sensing data in a cloud-computing environment.

A Novel Approach in Improving I/O Performance of Small Meteorological Files on HDFS

A Novel Scalable Architecture of Cloud Storage System for Small Files Based on P2P

A Novel Approach to Improving the Efficiency of Storing and Accessing Small Files on Hadoop: A Case Study by PowerPoint Files

Research on Data Cache Algorithm of Fog Computing Node

An archive‐based method for efficiently handling small file problems in HDFS

Storage-Optimization Method for Massive Small Files of Agricultural Resources Based on Hadoop

A Packaging Approach for Massive Amounts of Small Geospatial Files with HDFS.

A Proposed Approach for Improving Hadoop Performance for Handling Small Files

Improving Downloading Performance in Hadoop Distributed File System

Quick Compression and Transmission of Meteorological Big Data in Complicated Visualization Systems

A Numerical Model Oriented Large-scale Parallel I/O Optimization Method.

A Method for Optimizing Storage Efficiency of Meteorological Data in CIMISS

Meteorological data layout and task scheduling in a multi-cloud environment

A High Performance Compression Method For Climate

Cloud Storage of Massive Remote Sensing Data Based on Distributed File System

Addressing the Small Files Issue in Hadoop

A Novel Approach for Improving Security and Storage Efficiency on HDFS

Determine the Hardware Choice to Improve HDFS Performance Deployed in a Commodity Cluster

A distributed storage method of remote sensing data based on image blocks organization

CSFC: A New Centroid Based Clustering Method to Improve the Efficiency of Storing and Accessing Small Files in Hadoop

An Efficient Organization Method for Large-Scale and Long Time-Series Remote Sensing Data in a Cloud Computing Environment