Abstract:High-performance computing is progressively assuming a fundamental role in advancing scientific research and engineering domains. However, the ever-expanding scales of scientific simulations pose challenges for efficient data I/O and storage. The data compression technology has garnered significant attention as a solution to reduce data transmission and storage costs while enhancing performance. In particular, the BZIP2 lossless compression algorithm has been widely used due to its exceptional compression ratio, moderate compression speed, high reliability, and open-source nature. This paper focuses on the design and realization of a parallelized BZIP2 algorithm tailored for deployment on the New-Generation Sunway supercomputing platform. By leveraging the unique cache patterns of the New-Generation Sunway processor, we propose the highly tuned multi-threading and multi-node implementations of the BZIP2 applications for different scenarios. Moreover, we also propose the efficient BZIP2 libraries based on the management processing element and computing processing element which support the commonly used high-level (de)compression interfaces. The test results indicate that the our multi-threading implementation achieves maximum speedup of 23.09x$$ \times $$ (8.57x$$ \times $$) in decompression(compression) compared to the sequential implementation. Furthermore, the multi-node implementation achieves 50.81% (26.35%) parallel efficiency and peak performance of 16.6 GB/s (52.8 GB/s) for compression(decompression) when scaling up to 2048 processes. This paper focuses on the design and realization of a parallelized BZIP2 algorithm tailored for deployment on the New-Generation Sunway supercomputing platform. By leveraging the unique cache patterns of the New-Generation Sunway processor, we propose the highly tuned multi-threading and multi-node implementations of the BZIP2 applications for different scenarios. Moreover, we also propose the efficient BZIP2 libraries based on the management processing element and computing processing element which support the commonly used high-level (de)compression interfaces.image

ROOT I/O compression algorithms and their performance impact within Run 3

Increasing Parallelism in the ROOT I/O Subsystem

Evolution of the ROOT Tree I/O

Developments in ROOT I/O and trees

ROOT for the HL-LHC: data format

Software Challenges For HL-LHC Data Analysis

ROOT Status and Future Developments

Prototyping a ROOT-based distributed analysis workflow for HL-LHC: the CMS use case

GPU-based reconstruction and data compression at ALICE during LHC Run 3

400Gbps benchmark of XRootD HTTP-TPC

RNTuple performance: Status and Outlook

Polynomial data compression for large-scale physics experiments

Refactoring BZIP2 on the New‐generation Sunway Supercomputer

The LHCb software and computing upgrade for Run 3: opportunities and challenges

I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis

SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors

IC-Data: Improving Compressed Data Processing in Hadoop.

Parallel Writing of Nested Data in Columnar Formats

FCBench: Cross-Domain Benchmarking of Lossless Compression for Floating-Point Data