Abstract:High-performance computing is progressively assuming a fundamental role in advancing scientific research and engineering domains. However, the ever-expanding scales of scientific simulations pose challenges for efficient data I/O and storage. The data compression technology has garnered significant attention as a solution to reduce data transmission and storage costs while enhancing performance. In particular, the BZIP2 lossless compression algorithm has been widely used due to its exceptional compression ratio, moderate compression speed, high reliability, and open-source nature. This paper focuses on the design and realization of a parallelized BZIP2 algorithm tailored for deployment on the New-Generation Sunway supercomputing platform. By leveraging the unique cache patterns of the New-Generation Sunway processor, we propose the highly tuned multi-threading and multi-node implementations of the BZIP2 applications for different scenarios. Moreover, we also propose the efficient BZIP2 libraries based on the management processing element and computing processing element which support the commonly used high-level (de)compression interfaces. The test results indicate that the our multi-threading implementation achieves maximum speedup of 23.09x$$ \times $$ (8.57x$$ \times $$) in decompression(compression) compared to the sequential implementation. Furthermore, the multi-node implementation achieves 50.81% (26.35%) parallel efficiency and peak performance of 16.6 GB/s (52.8 GB/s) for compression(decompression) when scaling up to 2048 processes. This paper focuses on the design and realization of a parallelized BZIP2 algorithm tailored for deployment on the New-Generation Sunway supercomputing platform. By leveraging the unique cache patterns of the New-Generation Sunway processor, we propose the highly tuned multi-threading and multi-node implementations of the BZIP2 applications for different scenarios. Moreover, we also propose the efficient BZIP2 libraries based on the management processing element and computing processing element which support the commonly used high-level (de)compression interfaces.image

Leveraging Page-Level Compression In Mysql -A Practice At Baidu

Optimization Factor Analysis Of Large-Scale Join Queries On Different Platforms

Schema-Driven Performance Evaluation for Highly Concurrent Scenarios.

BushStore: Efficient B+Tree Group Indexing for LSM-Tree in Non-Volatile Memory

CompressDB: Enabling Efficient Compressed Data Direct Processing for Various Databases

Closing the B-tree vs. LSM-tree Write Amplification Gap on Modern Storage Hardware with Built-in Transparent Compression

Endurable SSD-Based Read Cache for Improving the Performance of Selective Restore from Deduplication Systems

Compressed Data Direct Computing for Databases

ZipCache: A DRAM/SSD Cache with Built-in Transparent Compression

Author Response for "refactoring BZIP2 on the New-Generation Sunway Supercomputer"

BlueJay: A Platform to Quantifying the Impact of Memory Latency on Datacenter Application Performance

FPGA-Accelerated Compactions for LSM-based Key-Value Store.

<i>SA-LSM</i>: Optimize Data Layout for LSM-tree Based Storage using Survival Analysis

Blitzcrank: Fast Semantic Compression for In-memory Online Transaction Processing

High-Ratio Compression for Machine-Generated Data

Optimizing NVMe Storage for Large-scale Deployment: Key Technologies and Strategies in Alibaba Cloud

Data Compression for Analytics over Large-scale In-memory Column Databases

Upscaledb: Efficient Integer-Key Compression in a Key-Value Store using SIMD Instructions

Characterizing and Optimizing TPC-C Workloads on Large-Scale Systems Using SSD Arrays

A Novel Optimization Method to Improve De-duplication Storage System Performance