Abstract:With the ever-increasing execution scale of high performance computing (HPC) applications, vast amounts of data are being produced by scientific research every day. Error-bounded lossy compression has been considered a very promising solution to address the big-data issue for scientific applications because it can significantly reduce the data volume with low time cost meanwhile allowing users to control the compression errors with a specified error bound. The existing error-bounded lossy compressors, however, are all developed based on inflexible designs or compression pipelines, which cannot adapt to diverse compression quality requirements/metrics favored by different application users. In this paper, we propose a novel dynamic quality metric oriented error-bounded lossy compression framework, namely QoZ. The detailed contribution is three-fold. (1) We design a novel highly-parameterized multi-level interpolation-based data predictor, which can significantly improve the overall compression quality with the same compressed size. (2) We design the error-bounded lossy compression framework QoZ based on the adaptive predictor, which can auto-tune the critical parameters and optimize the compression result according to user-specified quality metrics during online compression. (3) We evaluate QoZ carefully by comparing its compression quality with multiple state-of-the-arts on various real-world scientific application datasets. Experiments show that, compared with the second-best lossy compressor, QoZ can achieve up to 70% compression ratio improvement under the same error bound, up to 150% compression ratio improvement under the same PSNR, or up to 270% compression ratio improvement under the same SSIM.

Lessons Learned on the Path to Guaranteeing the Error Bound in Lossy Quantizers

Information loss and run time from practical application of quantum data compression

FRaZ: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework for Scientific Floating-point Data

NeurLZ: On Enhancing Lossy Compression Performance based on Error-Controlled Neural Learning for Scientific Data

A Framework for Robust Lossy Compression of Heavy-Tailed Sources

Use cases of lossy compression for floating-point data in scientific data sets

A Survey on Error-Bounded Lossy Compression for Scientific Datasets

Learnable Companding Quantization for Accurate Low-bit Neural Networks

Understanding The Effectiveness of Lossy Compression in Machine Learning Training Sets

cuSZ-$i$: High-Ratio Scientific Lossy Compression on GPUs with Optimized Multi-Level Interpolation

An Error-Bounded Lossy Compression Method with Bit-Adaptive Quantization for Particle Data

Error-aware Quantization through Noise Tempering

A Study On Quantization Effects Of Dct Based Compression

Black-Box Statistical Prediction of Lossy Compression Ratios for Scientific Data

LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit

Compression for Better: A General and Stable Lossless Compression Framework

QPET: A Versatile and Portable Quantity-of-Interest-preservation Framework for Error-Bounded Lossy Compression

A Lossy Method for Compressing Raw CCD Images

Lossy Compression for Schrödinger-style Quantum Simulations

Dynamic Quality Metric Oriented Error-bounded Lossy Compression for Scientific Datasets

On the best lattice quantizers