Abstract:High-performance computing is progressively assuming a fundamental role in advancing scientific research and engineering domains. However, the ever-expanding scales of scientific simulations pose challenges for efficient data I/O and storage. The data compression technology has garnered significant attention as a solution to reduce data transmission and storage costs while enhancing performance. In particular, the BZIP2 lossless compression algorithm has been widely used due to its exceptional compression ratio, moderate compression speed, high reliability, and open-source nature. This paper focuses on the design and realization of a parallelized BZIP2 algorithm tailored for deployment on the New-Generation Sunway supercomputing platform. By leveraging the unique cache patterns of the New-Generation Sunway processor, we propose the highly tuned multi-threading and multi-node implementations of the BZIP2 applications for different scenarios. Moreover, we also propose the efficient BZIP2 libraries based on the management processing element and computing processing element which support the commonly used high-level (de)compression interfaces. The test results indicate that the our multi-threading implementation achieves maximum speedup of 23.09x$$ \times $$ (8.57x$$ \times $$) in decompression(compression) compared to the sequential implementation. Furthermore, the multi-node implementation achieves 50.81% (26.35%) parallel efficiency and peak performance of 16.6 GB/s (52.8 GB/s) for compression(decompression) when scaling up to 2048 processes. This paper focuses on the design and realization of a parallelized BZIP2 algorithm tailored for deployment on the New-Generation Sunway supercomputing platform. By leveraging the unique cache patterns of the New-Generation Sunway processor, we propose the highly tuned multi-threading and multi-node implementations of the BZIP2 applications for different scenarios. Moreover, we also propose the efficient BZIP2 libraries based on the management processing element and computing processing element which support the commonly used high-level (de)compression interfaces.image

Dynamic Virtual Chunks: On Supporting Efficient Accesses to Compressed Scientific Data.

Data Compression and Storage under High Speed Network

Enabling Efficient Random Access to Hierarchically-Compressed Data

Scalable Hybrid Learning Techniques for Scientific Data Compression

A Versatile Compression Method for Floating-Point Data Stream

To Compress or Not To Compress: Energy Trade-Offs and Benefits of Lossy Compressed I/O

Dynamic image data compression in spatial and temporal domains: theory and algorithm

SDC Resilient Error-bounded Lossy Compressor

A High Performance Compression Method For Climate

Fast Compressed Segmentation Volumes for Scientific Visualization

A General Framework for Progressive Data Compression and Retrieval

Attention Based Machine Learning Methods for Data Reduction with Guaranteed Error Bounds

A self-aware data compression system on FPGA in Hadoop

Parallel Data Compression Techniques

Efficient Document Analytics on Compressed Data

AMRIC: A Novel In Situ Lossy Compression Framework for Efficient I/O in Adaptive Mesh Refinement Applications

IC-Data: Improving Compressed Data Processing in Hadoop.

Author Response for "refactoring BZIP2 on the New-Generation Sunway Supercomputer"

Use cases of lossy compression for floating-point data in scientific data sets

Dynamic Quality Metric Oriented Error-bounded Lossy Compression for Scientific Datasets

MetaZip