A Versatile Compression Method for Floating-Point Data Stream

Songbin Liu,Xiaomeng Huang,Yufang Ni,Haohuan Fu,Guangwen Yang

DOI: https://doi.org/10.1109/icndc.2013.32

2013-01-01

Abstract:With the rapid advances in supercomputing and numerical simulations, the output data of scientific computing is expanding rapidly, bringing tough challenges for data sharing and data archiving. Data compression can mitigate these challenges by reducing the size of the data to be stored or transferred. However, data compression has to achieve a good balance between compression ratios and throughput, before it can be employed in the high-end computing environments. In this paper, we propose and evaluate a versatile compression method for floating-point data. Firstly, it can achieve much better compression ratios than existing general purpose compression methods with promising throughputs. Secondly, it supports asymmetric decompression: losslessly compressed data can be decompressed lossily, thus facilitating data analysis in different precision requirements. Thirdly, it can leverage existing different kinds of general purpose compressors (zlib, lz4, for instance), and provide more flexible trade-offs between compression ratios and throughputs. Evaluations demonstrate that our compressor can achieve comparable compression ratios with the best compressors, while the compression and decompression throughputs can be 10 times higher than them. The single thread compression throughputs can be 135 MB/s, and the decompression throughputs can be 194 MB/s.