Abstract:In scientific simulations, observations, and experiments, the cost of transferring data to and from disk and across networks has become a significant bottleneck that particularly impacts subsequent data analysis and visualization. To address this challenge, compression techniques have been widely adopted. However, traditional lossy compression approaches often require setting error tolerances conservatively to respect the numerical sensitivities of a wide variety of post hoc data analyses, some of which may not even be known a priori. Progressive data compression and retrieval has emerged as a solution, allowing for the adaptive handling of compressed data according to the needs of a given post-processing task. However, few analysis algorithms natively support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. This work presents a general framework that supports progressive-precision data queries independently of the underlying data compressor or number representation. Our approach is based on a multiple-component representation that successively, with each new component, reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented our approach on top of four popular scientific data compressors and have evaluated its behavior on several real-world data sets from the SDRBench collection. Numerical results indicate that our framework is effective in terms of accuracy compared to each of the standalone compressors it builds upon. In addition, (de)compression time is proportional to the number and granularity of components. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

Enabling Random Access in Universal Compressors

Enabling Efficient Random Access to Hierarchically-Compressed Data

Reinforcement Learning Based Data Compression for Energy-Efficient Non-volatile Caches.

MetaZip

A Versatile Compression Method for Floating-Point Data Stream

Enabling Efficient Random Access to Hierarchically Compressed Text Data on Diverse GPU Platforms

ZipCache: A DRAM/SSD Cache with Built-in Transparent Compression

CMIC: an efficient quality score compressor with random access functionality

CompressDB: Enabling Efficient Compressed Data Direct Processing for Various Databases

Data Compaction - Compression without Decompression

A General Framework for Progressive Data Compression and Retrieval

SDC Resilient Error-bounded Lossy Compressor

Time-universal data compression and prediction

To Compress or Not To Compress: Energy Trade-Offs and Benefits of Lossy Compressed I/O

Parallel Data Compression Techniques

MetaZip: a high-throughput and efficient accelerator for DEFLATE

Efficient Compression Technique for Sparse Sets

Compressed Data Direct Computing for Databases

C-Pack: A High-Performance Microprocessor Cache Compression Algorithm

Czip: A Fast Lossless Compression Algorithm for Climate Data

RAGE for the Machine: Image Compression with Low-Cost Random Access for Embedded Applications