Abstract:In scientific simulations, observations, and experiments, the cost of transferring data to and from disk and across networks has become a significant bottleneck that particularly impacts subsequent data analysis and visualization. To address this challenge, compression techniques have been widely adopted. However, traditional lossy compression approaches often require setting error tolerances conservatively to respect the numerical sensitivities of a wide variety of post hoc data analyses, some of which may not even be known a priori. Progressive data compression and retrieval has emerged as a solution, allowing for the adaptive handling of compressed data according to the needs of a given post-processing task. However, few analysis algorithms natively support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. This work presents a general framework that supports progressive-precision data queries independently of the underlying data compressor or number representation. Our approach is based on a multiple-component representation that successively, with each new component, reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented our approach on top of four popular scientific data compressors and have evaluated its behavior on several real-world data sets from the SDRBench collection. Numerical results indicate that our framework is effective in terms of accuracy compared to each of the standalone compressors it builds upon. In addition, (de)compression time is proportional to the number and granularity of components. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

Seismic data compression: an overview

Impact of Lossy Compression Errors on Passive Seismic Data Analyses

Seismic Data Compression Using Deep Learning

Some Practical Aspects of Seismic Data Compression Based on Wavelet Transform

Feature-Preserving Quantization for 3D Seismic Visualization

A Survey on Error-Bounded Lossy Compression for Scientific Datasets

SDC Resilient Error-bounded Lossy Compressor

A General Framework for Progressive Data Compression and Retrieval

Adaptive resampling for data compression

Efficient Document Analytics on Compressed Data

A Survey on Data Compression Techniques for Automotive LiDAR Point Clouds

Optimising the Processing and Storage of Visibilities using lossy compression

Compressing Repeated Content Within Large-Scale Remote Sensing Images

SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors

3DGS.zip: A survey on 3D Gaussian Splatting Compression Methods

A Quick Review on Recent Trends in 3D Point Cloud Data Compression Techniques and the Challenges of Direct Processing in 3D Compressed Domain

Spatiotemporally adaptive compression for scientific dataset with feature preservation -- a case study on simulation data with extreme climate events analysis

Dreamlet compression of seismic data

Dreamlet Transform Applied to Seismic Data Compression and Its Effects on Migration

Compression Supports Spatial Deep Learning

Scalable Hybrid Learning Techniques for Scientific Data Compression