Abstract:In scientific simulations, observations, and experiments, the cost of transferring data to and from disk and across networks has become a significant bottleneck that particularly impacts subsequent data analysis and visualization. To address this challenge, compression techniques have been widely adopted. However, traditional lossy compression approaches often require setting error tolerances conservatively to respect the numerical sensitivities of a wide variety of post hoc data analyses, some of which may not even be known a priori. Progressive data compression and retrieval has emerged as a solution, allowing for the adaptive handling of compressed data according to the needs of a given post-processing task. However, few analysis algorithms natively support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. This work presents a general framework that supports progressive-precision data queries independently of the underlying data compressor or number representation. Our approach is based on a multiple-component representation that successively, with each new component, reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented our approach on top of four popular scientific data compressors and have evaluated its behavior on several real-world data sets from the SDRBench collection. Numerical results indicate that our framework is effective in terms of accuracy compared to each of the standalone compressors it builds upon. In addition, (de)compression time is proportional to the number and granularity of components. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

Enabling near real-time remote search for fast transient events with lossy data compression

Optimising the Processing and Storage of Visibilities using lossy compression

Finding Fast Transients in Real Time Using Novel Light Curve Analysis Algorithm

Extending CCSDS 123.0-B-1 for Lossless 4D Image Compression

Comparative Study of Lossy and Lossless Data Compression in Distributed Optical Fiber Sensing Systems

The Deeper Wider Faster program: chasing the fastest bursts in the Universe

Lossy Compression of Large-Scale Radio Interferometric Data

Investigation of fast and efficient lossless compression algorithms for macromolecular crystallography experiments

Astronomical Imagery: Considerations For a Contemporary Approach with JPEG2000

Optimising the Processing and Storage of Radio Astronomy Data

Massive Compression for High Data Rate Macromolecular Crystallography (HDRMX): Impact on Diffraction Data and Subsequent Structural Analysis

Polynomial data compression for large-scale physics experiments

Lossy compression of infrared videos in WEST (W Environment in Steady-state Tokamak)

Decomposed Soft Compression for Remote Sensing Image.

Streaming Large-Scale Electron Microscopy Data to a Supercomputing Facility

AstroDLLC: Efficiently Reducing Storage and Transmission Costs for Massive Solar Observation Data via Deep Learning-based Lossless Compression

ReCoDe: A Data Reduction and Compression Description for High Throughput Time-Resolved Electron Microscopy

A General Framework for Progressive Data Compression and Retrieval

Gpu-Based Lossless Compression Of Aurora Spectral Data Using Online Dpcm

Improving the Scalability of Hyperspectral Imaging Applications on Heterogeneous Platforms Using Adaptive Run-Time Data Compression

Task-Oriented Compression Framework for Remote Sensing Satellite Data Transmission