Abstract:Hyperspectral infrared atmospheric sounding data, characterized by their high vertical resolution, play a crucial role in capturing three-dimensional atmospheric spatial information. The hyperspectral infrared atmospheric detectors HIRAS/HIRAS-II, mounted on the FY3D/EF satellite, have established an initial global coverage network for atmospheric sounding. The collaborative observation approach involving multiple satellites will improve both the coverage and responsiveness of data acquisition, thereby enhancing the overall quality and reliability of the data. In response to the increasing number of channels, the rapid growth of data volume, and the specific requirements of multi-satellite joint observation applications with infrared hyperspectral sounding data, this paper introduces an efficient storage and indexing method for infrared hyperspectral sounding data within a distributed architecture for the first time. The proposed approach, built on the Kubernetes cloud platform, utilizes the Google S2 discrete grid spatial indexing algorithm to establish a grid-based hierarchical model for unified metadata-embedded documents. Additionally, it optimizes the rowkey design using the BPDS model, thereby enabling the distributed storage of data in HBase. The experimental results demonstrate that the query efficiency of the Google S2 grid-based embedded document model is superior to that of the traditional flat model, achieving a query time that is only 35.6% of the latter for a dataset of 5 million records. Additionally, this method exhibits better data distribution characteristics within the global grid compared to the H3 algorithm. Leveraging the BPDS model, the HBase distributed storage system adeptly balances the node load and counteracts the detrimental effects caused by the accumulation of time-series remote sensing images. This architecture significantly enhances both storage and query efficiency, thus laying a robust foundation for forthcoming distributed computing.

Efficient Indexing and Querying of Massive Astronomical Data Using Compressed Word-Aligned Hybrid Bitmap

Massive astronomical real-time data indexing based on compressed word-aligned bitmap

NVST DATA ARCHIVING SYSTEM BASED ON FASTBIT NOSQL DATABASE

High-Performance Data Storage for Large Solar Telescope Using Distributed File System

Low-cost high performance distributed data storage for multi-channel observations

Adaptive Data Compression for Massive Scan Data

Using Bitmap Index to Accelerate Accessing Large Scale Scientific Data on Demand

Astronomical Data Preprocessing Implementation Based On Fpga And Data Transformation Strategy For The Fast Telescope As A Giant Cps

An Efficient and Compact Indexing Scheme for Large-Scale Data Store.

The Application of ElasticSearch in the Massive Astronomical Data Retrieval

Distribution-Based Approach for Efficient Storage and Indexing of Massive Infrared Hyperspectral Sounding Data

Quick-read Means for Large Scale Images Based on Storage Mapping File Technique

AstroServ: Distributed Database for Serving Large-Scale Full Life-Cycle Astronomical Data

AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression

Enhanced Management of Personal Astronomical Data with FITSManager

HBaseSpatial: A Scalable Spatial Data Storage Based on HBase

High Performance Negative Database for Massive Data Management System of The Mingantu Spectral Radioheliograph

Efficient Management and Scheduling of Massive Remote Sensing Image Datasets

Research on metadata storage and querying of FITS file based on NoSQL

"Superluminal" FITS File Processing on Multiprocessors: Zero Time Endian Conversion Technique

Parallel data analysis directly on scientific file formats