Abstract:In scientific simulations, observations, and experiments, the cost of transferring data to and from disk and across networks has become a significant bottleneck that particularly impacts subsequent data analysis and visualization. To address this challenge, compression techniques have been widely adopted. However, traditional lossy compression approaches often require setting error tolerances conservatively to respect the numerical sensitivities of a wide variety of post hoc data analyses, some of which may not even be known a priori. Progressive data compression and retrieval has emerged as a solution, allowing for the adaptive handling of compressed data according to the needs of a given post-processing task. However, few analysis algorithms natively support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. This work presents a general framework that supports progressive-precision data queries independently of the underlying data compressor or number representation. Our approach is based on a multiple-component representation that successively, with each new component, reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented our approach on top of four popular scientific data compressors and have evaluated its behavior on several real-world data sets from the SDRBench collection. Numerical results indicate that our framework is effective in terms of accuracy compared to each of the standalone compressors it builds upon. In addition, (de)compression time is proportional to the number and granularity of components. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

A High Performance Compression Method For Climate

Czip: A Fast Lossless Compression Algorithm for Climate Data

Integrating Online Compression to Accelerate Large-Scale Data Analytics Applications

Data Reduction Analysis for Climate Data Sets

Compression and Conditional Emulation of Climate Model Output

Spatiotemporally adaptive compression for scientific dataset with feature preservation -- a case study on simulation data with extreme climate events analysis

A Versatile Compression Method for Floating-Point Data Stream

Lossy Compression of Earth System Model Data Based on a Hierarchical Tensor with Adaptive-HGFDR (V1.0)

High Performance Lossless Compression of Scientific Floating Data

Quasi-lossless Compression Encoding for Meteorological Grid Data

Compressing Repeated Content Within Large-Scale Remote Sensing Images

A Framework for Automatic Validation and Application of Lossy Data Compression in Ensemble Data Assimilation

HiHa: Introducing Hierarchical Harmonic Decomposition to Implicit Neural Compression for Atmospheric Data

Fast Multidimensional Ensemble Empirical Mode Decomposition For The Analysis Of Big Spatio-Temporal Datasets

Accelerating Lossy Compression on HPC Datasets Via Partitioning Computation for Parallel Processing

FCBench: Cross-Domain Benchmarking of Lossless Compression for Floating-Point Data

Accelerating Relative-error Bounded Lossy Compression for HPC Datasets with Precomputation-Based Mechanisms.

AMP: Total Variation Reduction for Lossless Compression Via Approximate Median-based Preconditioning

Total Variation Reduction for Lossless Compression of HPC Applications

CRA5: Extreme Compression of ERA5 for Portable Global Climate and Weather Research via an Efficient Variational Transformer

A General Framework for Progressive Data Compression and Retrieval