Design and Optimization of Zstandard Algorithm Based on Concurrent Streaming of Multiple Hash Tables
Leidong Zheng,Youyu Wu,Min Zhu,Wei Ge,jinjiang Yang
DOI: https://doi.org/10.1117/12.2649516
2022-01-01
Abstract:When compressing massive data, using software implementation of Zstandard (Zstd) algorithm will occupy a lot of processor resources, and it will lead to performance bottleneck due to serial data dependency, hardware acceleration of Zstd algorithm is an effective solution to the above problems. Therefore, in this paper, we propose a hardware architecture for concurrent streaming of multiple hash tables applicable to the Zstd algorithm. Using this design scheme, the RTL-based VCS simulation verification shows that the compression ratio almost reaches the standard of software, the compression speed of hardware is about 5.5 times of software compression speed, and the decompression speed of hardware is about 5 times of software, where the compression speed reaches up to 1.17GB/s and decompression speed reaches up to 1.89GB/s, and compared with the existing hardware implementation of Zstd, the compression speed is about 11.4% higher than that of the existing hardware implementation of Zstd.