Abstract:Hyperspectral infrared atmospheric sounding data, characterized by their high vertical resolution, play a crucial role in capturing three-dimensional atmospheric spatial information. The hyperspectral infrared atmospheric detectors HIRAS/HIRAS-II, mounted on the FY3D/EF satellite, have established an initial global coverage network for atmospheric sounding. The collaborative observation approach involving multiple satellites will improve both the coverage and responsiveness of data acquisition, thereby enhancing the overall quality and reliability of the data. In response to the increasing number of channels, the rapid growth of data volume, and the specific requirements of multi-satellite joint observation applications with infrared hyperspectral sounding data, this paper introduces an efficient storage and indexing method for infrared hyperspectral sounding data within a distributed architecture for the first time. The proposed approach, built on the Kubernetes cloud platform, utilizes the Google S2 discrete grid spatial indexing algorithm to establish a grid-based hierarchical model for unified metadata-embedded documents. Additionally, it optimizes the rowkey design using the BPDS model, thereby enabling the distributed storage of data in HBase. The experimental results demonstrate that the query efficiency of the Google S2 grid-based embedded document model is superior to that of the traditional flat model, achieving a query time that is only 35.6% of the latter for a dataset of 5 million records. Additionally, this method exhibits better data distribution characteristics within the global grid compared to the H3 algorithm. Leveraging the BPDS model, the HBase distributed storage system adeptly balances the node load and counteracts the detrimental effects caused by the accumulation of time-series remote sensing images. This architecture significantly enhances both storage and query efficiency, thus laying a robust foundation for forthcoming distributed computing.

Multi-Dimensional Index Over A Key-Value Store For Semi-Structured Data

An Efficient and Compact Indexing Scheme for Large-Scale Data Store.

Supporting Database Applications As a Service

A Survey of Multi-Dimensional Indexes: Past and Future Trends

Indexing multi-dimensional data in a cloud system.

Scalable Top-K Spatial Keyword Search

HBaseSpatial: A Scalable Spatial Data Storage Based on HBase

MTDB: an LSM-tree-based key-value store using a multi-tree structure to improve read performance

Using Bitmap Index to Accelerate Accessing Large Scale Scientific Data on Demand

MC-Tree: Dynamic Index Structure for Partially Clustered Multi-Dimensional Database

HiStore: Rethinking Hybrid Index in RDMA-based Key-Value Store

Research on Dynamic Indexing Structure for Multi-Dimensional Vectors

Efficient Privacy-Preserving Multi-Dimensional Range Query for Cloud-Assisted Ehealth Systems

Hierarchical Bitmap Indexing for Range and Membership Queries on Multidimensional Arrays

A Mapping Based Approach for Multidimensional Data Indexing

A Hybrid Index Structure Based on Multi-Core Cluster

Vector Spatial Big Data Storage and Optimized Query Based on the Multi-Level Hilbert Grid Index in HBase

Distribution-Based Approach for Efficient Storage and Indexing of Massive Infrared Hyperspectral Sounding Data

Multidimensional data index method of energy big data center based on space filling curve

HMIBase： an Hierarchical Indexing System for Storing and Querying Big Data

Bf-Matrix: A Secondary Index For The Cloud Storage