Fpga Implementation Of A Universal Lossless Data Compressor

J Hong,Lj Meng,J Zu

2005-01-01

Abstract:The paper introduced the authors' research work on an universal lossless data compressor This compressor has been used in many storage data acquisition systems. The compression algorithm used in this paper was an improved LZW. The paper also includes algorithm's VHDL simulation results and the FPGA implementation of it.

What problem does this paper attempt to address?

Data Compression and Storage under High Speed Network

ZHANG Bao-jun,PING Ling-di,PAN Xue-zeng,WANG Jie-bing

DOI: https://doi.org/10.3785/j.issn.1008-973x.2009.09.007

2009-01-01

Abstract:A practical scheme was proposed aiming at the problem of data compression and storage under high speed network.The scheme compressed the data with hardware and software,and without any performance affection to the network speed.HIFN data reduction(DR) series high performance lossless data compression card was used in hardware.The card had multi 9630 compression chips and used the eLZS compression algorithm to provide fast data compression and decompression with satisfactory compression ratio.An eLZS module embedded in the DR device driver compressed and decompressed the data seamlessly when the hardware was failure in order to make the system run normally.The Windows driver of DR was developed in order to make DR compression card work on Windows effectively.The scheme was realized on both Linux and Windows platform and showed good performance under the test.
Multi-core Lossless Compression System for High-speed Data and Its FPGA Implementation

HAN Kai,ZHAO Si-cong,ZHANG Li,WU Biao

2013-01-01

Abstract:In the fields of aviation remote sensing image transmission and radar data storage,the requirements of data quality and real-time performance are critical.A multi-core lossless compression system solution for high-speed data was introduced in this paper.This solution used the improved LZW algorithm which is as more than ten times fast as the LZW algorithm.This thesis used Xilinx's Spartan6 LX150T FPGA as the hardware platform and completed the compression system which has sixteen encoders.Experiments show that the bandwidth of the single FPGA compress module is about 1.1Gbps when the frequency of system clock is 100MHz.This article put forward the design proposal of compress system containing five FPGAs which has the bandwidth of 4Gbps.
A Hardware Implementation of Real Time Lossless Data Compression and Decompression Circuits

Le Yang,Zhao Yang Guo,Shan,Feng Guo,Xin An Wang

DOI: https://doi.org/10.4028/www.scientific.net/amm.719-720.554

2015-01-01

Applied Mechanics and Materials

Abstract:This paper presents a hardware implementation of real time data compression and decompression circuits based on the LZW algorithm. LZW is a dictionary based data compression, which has the advantage of fast speed, high compression, and small resource occupation. In compression circuit, the design creatively utilizes two dictionaries alternately to improve efficiency and compressing rate. In decompression circuit, an integrated State machine control module is adopted to save hardware resource. Through hardware description and language programming, the circuits finally reach function simulation and timing simulation. The width of data sample is 12bits, and the dictionary storage capacity is 1K. The simulation results show the compression and decompression circuits have complete function. Compared to software method, hardware implementation can save more storage and compressing time. It has a high practical value in the future.
FPGA bitstream compression and decompression using LZ and golomb coding (abstract only).

Jinsong Mao,Hao Zhou,Haijiang Ye,Jinmei Lai

DOI: https://doi.org/10.1145/2435264.2435309

2013-01-01

Abstract:In this paper we propose an optimized bitstream compression algorithm based on LZ and a novel architecture of decompressor, the proposed algorithm improves the Compression Ratio by fully utilizing the regularity of configuration bits of CLB (Configurable Logic Box) in FPGA and using the variable length Golomb coding method. The experimental results show that the Optimized method can improve the Compression Ratio of LZSS by 32.3% for bitstream with high regularity and 10.3% for bitstream with low regularity, and our approach shows a higher flexibility than the BMC+RLE arithmetic when compressing the bitstream with high regularity for various FPGA. Moreover, we design a two-buffer-window decompressor to download the compressed bitstreams. In order to increase the throughput of the proposed decompressor, we design a multi-stage data selector in it. The post-simulation of the decompressor shows that its throughput is up to 9280 Mbps under 65nm CMOS process. And that is 4352Mbps when verified on a Virtex-5 FPGA.
An Offloading Architecture of Lossless Compression Based on Smart NIC

Can Zhang,Yaming Xu,Hongliang Wang,Wei Liu,Qi Mu,Wei Guo

DOI: https://doi.org/10.1109/icceic51584.2020.00049

2020-01-01

Abstract:With the explosive growth of information, data transmission and compression are becoming more and more important. Most of the existing compression algorithms are implemented by software architecture, which results in the waste of CPU resources. By unloading the algorithm to hardware, it is beneficial to reasonably allocate resources and improve the processing efficiency of CPU. The smart Network Interface Cards (NIC) based on FPGA can not only complete the task of data unloading, but also have the function of sending the compressed data to the cloud storage device directly. It provides a new operating environment for the compression algorithm with efficient network transmission speed, and is able to reduce the system power consumption within a certain range.In this paper, we design an FPGA-based smart NIC that has the ability to unload CPU’s LZ4 compression algorithm to FPGA to improve the utilization rate of CPU. Experiments show that the architecture is able to successfully open up the data path, and the network speed is nearly 46 Gb/s. At the same time, it can be seen from the experiments that the time required to run the LZ4 compression algorithm on the CPU is proportional to the size of files. It shows that for large files, using a smart NIC for data offloading has strong practical significance.
Fpga Bitstream Compression and Decompression Based on Lz77 Algorithm and Bmc Technique

Yuanpei Gao,Haijiang Ye,Jian Wang,Jinmei Lai

DOI: https://doi.org/10.1109/asicon.2015.7517171

2015-01-01

Abstract:In this paper, we propose a FPGA bitstream compression method based on LZ77 algorithm and Bitmask-based Compression(BMC) Technique. In order to improve the compression ratio, we optimize LZ77 by encoding the variable 'length' with golomb code and simplifying the encoding output. We develop appropriate adding strategy and choose proper cost-benefit parameters of BMC, and combine the BMC with LZ77 to further improve the compression ratio. Experiment results shows that our method has 12.9% improvement in compression ratio over fixed dictionary approach and 9.3% over LZSS on an average. Our compression method also has good compatibility. A decompressor which has two look-ahead buffers is designed for decompression. The pipeline architecture of the decompressor improves the throughput. The decompression throughput reaches 1692Mb/s according to Xilinx ISE report. We have verified the decompressor on a Virtex-5 FPGA.
Method Of Lfm Pulse Compression Implementation Based On Fpga

Fu Ning,Wang Yuze,Xu Hongwei,Qiao Liyan,Jin Hong

DOI: https://doi.org/10.1109/ICEMI.2013.6743104

2013-01-01

Abstract:For the achievement of LFM signal acquisition, pulse compression and storage, the hardware platform is built. A way to implement the pulse compression of the LFM signal based on FPGA is presented. This paper describes in detail the functions and implementation methods of the various functional modules of the FPGA in the pulse compression process. A theoretical simulation on the pulse compression is done, and the actual test results are analyzed after compared with the theoretical simulation.
UH-JLS: A Parallel Ultra-High Throughput JPEG-LS Encoding Architecture for Lossless Image Compression

Xuan Wang,Lei Gong,Chao Wang,Xi Li,Xuehai Zhou

DOI: https://doi.org/10.1109/iccd53106.2021.00060

2021-01-01

Abstract:The lossless image compression technique has a great application value in distortion-sensitive applications. JPEG-LS, as a mature lossless compression standard, is widely adopted for its excellent compression ratio. Many hardware JPEG-LS compressors are proposed on FPGAs and ASICs to achieve high energy efficiency and low cost. However, JPEG-LS has a contextual Read-After-Write (RAW) issue, making previous hardware either insufficiently explore its parallelism potential or induce other defects while parallelizing, such as compression ratio dropping and compatibility problems. In this paper, we propose a hardware/software co-design method for high-performance JPEG-LS compressor design. At the software level, we propose a pixel grouping scheduling scheme and the Pseudo-LS method to tap the parallelism aiming at the RAW issue. At the hardware level, we discuss the high-performance design methods of these software-level schemes and propose a design space exploration method to constrain the resource usage introduced by parallelization. To our knowledge, our architecture, UH-JLS, is the first pixel-level parallelization streaming image compressor based on the standard JPEG-LS. The experiments show that in the lossless manner and the Pseudo-LS manner, UH-JLS respectively achieves 5.6x and 7.1x speedup than the previous state-of-the-art FPGA-based JPEG-LS compressor.
An FPGA-Based LOCO-ANS Implementation for Lossless and Near-Lossless Image Compression Using High-Level Synthesis

Tobías Alonso,Gustavo Sutter,Jorge E. López de Vergara

DOI: https://doi.org/10.3390/electronics10232934

IF: 2.9

2021-11-26

Electronics

Abstract:In this work, we present and evaluate a hardware architecture for the LOCO-ANS (Low Complexity Lossless Compression with Asymmetric Numeral Systems) lossless and near-lossless image compressor, which is based on JPEG-LS standard. The design is implemented in two FPGA generations, evaluating its performance for different codec configurations. The tests show that the design is capable of up to 40.5 MPixels/s and 124 MPixels/s per lane for Zynq 7020 and UltraScale+ FPGAs, respectively. Compared to the single thread LOCO-ANS software implementation running in a 1.2 GHz Raspberry Pi 3B, each hardware lane achieves 6.5 times higher throughput, even when implemented in an older and cost-optimized chip like the Zynq 7020. Results are also presented for a lossless only version, which achieves a lower footprint and approximately 50% higher performance than the version that supports both lossless and near-lossless. Interestingly, these great results were obtained applying High-Level Synthesis, describing the coder with C++ code, which tends to establish a trade-off between design time and quality of results. These results show that the algorithm is very suitable for hardware implementation. Moreover, the implemented system is faster and achieves higher compression than the best previously available near-lossless JPEG-LS hardware implementation.

engineering, electrical & electronic,computer science, information systems,physics, applied
FPGA Acceleration of Zstd Compression Algorithm

Jianyu Chen,Maurice Daverveldt,Zaid Al-Ars

DOI: https://doi.org/10.1109/ipdpsw52791.2021.00035

2021-01-01

Abstract:With the continued increase in the amount of big data generated and stored in various application domains, such as high-frequency trading, compression techniques are becoming ever more important to reduce the requirements on communication bandwidth and storage capacity. Zstandard (Zstd) is emerging as an important compression algorithm for big data sets capable of achieving a good compression ratio but with a higher speed than comparable algorithms. In this paper, we introduce the architecture of a new hardware compression kernel for Zstd that allows the algorithm to be used for real-time compression of big data streams. In addition, we optimize the proposed architecture for the specific use case of streaming high-frequency trading data. The optimized kernel is implemented on a Xilinx Alveo U200 board. Our optimized implementation allows us to fit ten kernel blocks on one board, which is able to achieve a compression throughput of about 8.6GB/s and compression ratio of about 23.6%. The hardware implementation is open source and publicly available at https://github.com/ChenJianyunp/Hardware-Zstd-Compression-Unit.
General Adaptive Lossless Compression for Multi-Channel Sensor Signals

Haori Zheng,Qianhao Chen,Feng Yu,Bo Zhang

DOI: https://doi.org/10.1109/tcsii.2023.3294504

2023-01-01

Abstract:The storage and transmission of multi-channel high-rate sensor data have emerged as new challenges in real-time sensor signal acquisition. To address these challenges, a multi-channel compression algorithm can be utilized to decrease data storage and transmission bandwidth requirements. We propose a general adaptive lossless multi-channel compression architecture that leverages inter-channel and intra-channel correlation to improve the compression efficiency. Additionally, a high-throughput hardware implementation of the compression architecture is proposed. The experimental results on multiple open-source datasets demonstrate that our multi-channel lossless compression architecture outperforms reported single-channel and multi-channel compression architectures in terms of compression efficiency. Experiments show that it can deliver a competitive throughput rate for each channel in FPGA for a 16-channel compression implementation.
Lossless Compression of Bitstream Configuration Files: Towards FPGA Cloud

Jinyu Wang,Yifei Kang,Yaqi Feng,Yiwen Li,Weiguo Wu,Guoliang Xing

DOI: https://doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00192

2021-01-01

Abstract:With the development of integrated circuit technology, the scale of Field Programmable Gate Array (FPGA) devices and the quantity of on-chip resources have been increased dramatically, as well as the size of configuration files, rendering the configuration process time-consuming. Especially, in the cloud-based multiple FPGAs co-designs, the transmission of large scale bitstream configuration files in public networks may suffer unpredictable delay, which causes circuits with high real-time requirements failed to be timely executed. To address this challenge, we divide a configuration file into three portions which have different data structures, and different compression methods are proposed to fit the data structure of each part to obtain a lower compression rate. For the portion which describes the FPGA on-chip resources, an LSTM and ID-CNN based arithmetic coding compression method is introduced. In this way, the compression problem is converted to an estimation of a conditional probability distribution problem. Finally, to improve the prediction efficiency of the neural network, a cache based neural network inference acceleration method is proposed. Our approach is validated on the standard benchmark data sets and the average compression rate is 28.89%. Compared with seven different compression methods, our approach improves the compression rate by up to 39.45%.
Adaptive Pipeline Hardware Architecture Design and Implementation for Image Lossless Compression/Decompression Based on JPEG-LS

Fangjia Liu,Xiyao Chen,Zijun Liao,Chong Yang

DOI: https://doi.org/10.1109/access.2024.3350344

IF: 3.9

2024-01-01

IEEE Access

Abstract:Lossless compression of medical images can reduce data size, save storage and transmission costs, and cope with the challenge of increasing resolution demand for medical images. This paper proposes an adaptive pipeline hardware architecture for lossless compression/decompression of medical images based on joint photographic experts group-lossless (JPEG-LS), which can achieve efficient image processing on field programmable gate array (FPGA). In addition, by adding a pipeline pause mechanism, the problem of coding errors caused by parameter updates is solved. On the one hand, the binary search method is used to optimize the calculation process of the Golomb coding parameter k in the pipeline, which reduces the parameter update delay. On the other hand, the construction process of the context causal template is optimized, which effectively improves the throughput. Based on the parallel block compression architecture, a hardware module for alpha red green blue (ARGB) image block compression/decompression with customizable block size is implemented, which improves the image compression/decompression throughput and reduces the compression and decompression delay. This paper implements the hardware architecture on Zynq7000 FPGA and evaluates its performance. Compared with existing schemes, the proposed architecture uses less resources and achieves higher compression ratio and clock frequency.

computer science, information systems,telecommunications,engineering, electrical & electronic
Lossless Compression Decoders for Bitstreams and Software Binaries Based on High-Level Synthesis.

Jian Yan,Junqi Yuan,Philip H. W. Leong,Wayne Luk,Lingli Wang

DOI: https://doi.org/10.1109/tvlsi.2017.2713527

2017-01-01

IEEE Transactions on Very Large Scale Integration (VLSI) Systems

Abstract:As the density of field-programmable gate arrays continues to increase, the size of configuration bitstreams grows accordingly. Compression techniques can reduce memory size and save external memory bandwidth. To accelerate the configuration process and reduce the software startup time, four open-source lossless compression decoders developed using high-level synthesis techniques are presented. Moreover, in order to balance the objectives of compression ratio, decompression throughput, and hardware resource overhead, various improvements and optimizations are proposed. Full bitstreams and software binaries have been collected as a benchmark, and 33 partial bitstreams have also been developed and integrated into the benchmark. Evaluations of the synthesizable compression decoders are demonstrated on a Xilinx ZC706 board, showing higher decompression throughput than those of the existing lossless compression decoders using our benchmark. The proposed decoders can reduce software startup time by up to 31.23% in embedded systems and 69.83% reduction of reconfiguration time for partial reconfigurable systems.
Research and Implementation of FPGA Image Compression Parallel Algorithm Based on CCSDS

Leqi Yang

DOI: https://doi.org/10.56028/aetr.11.1.572.2024

2024-07-18

Abstract:High-pixel, high-resolution spectral images are an important source of key graphical information obtained from space remote sensing, however, with the dramatically expanding amount of graphical information, the limited transmission bandwidth and hardware resources are difficult to meet the demand for massive data, which makes real-time image lossless compression and transmission technology an important solution. For this reason, this paper is based on the CCSDS image lossless compression algorithm, using FPGA for the construction of image compression system, and proposes a method strategy based on parallel acceleration algorithm to realize the real time processing of large-capacity hyperspectral images. The simulation results show that hyperspectral river and mountain range images can be compressed with greatly improved compression efficiency while maintaining real time performance and accuracy.
Design and Implementation of a Lossless Compression System for Hyperspectral Images

Qizhi Fang,Yuxuan Liu,Lili Zhang

DOI: https://doi.org/10.18280/ts.370506

IF: 2.299

2020-11-25

Traitement du signal

Abstract:Despite its popularity, the hyperspectral image compression algorithm recommended by the Consultative Committee for Space Data Systems (CCSDS) faces a long delay of the feedback loop and complex computations in the modes of band sequential (BSQ) and band interleaved by line (BIL). After analyzing the features of the CCSDS algorithm, this paper proposes a forward prediction method based on the xc7k325tffg9000 field programmable gate array (FPGA) chip (Xilinx Inc.), and adjusts the calculation flow of the CCSDS algorithm, aiming to shorten the time delay in the feedback loop. In addition, full-pipeline construction was implemented on FPGA board to realize real-time processing of data, and dynamic configuration of image parameters. Through functional simulation and off-board test, it is learned that, for the speed-insensitive path, the optimized algorithm can realize the complex operations of the original algorithm with less hardware resources; for hyperspectral image data with an effective input bit width of 12bit, the proposed method can reach a maximum operating frequency of 103MHz, and the data throughput of 103M samples per second (1.237Gbps).

computer science, artificial intelligence,engineering, electrical & electronic
From a Lossless (~1.5:1) Compression Algorithm for Llama2 7B Weights to Variable Precision, Variable Range, Compressed Numeric Data Types for CNNs and LLMs

Vincenzo Liguori

2024-04-17

Abstract:This paper starts with a simple lossless ~1.5:1 compression algorithm for the weights of the Large Language Model (LLM) Llama2 7B [1] that can be implemented in ~200 LUTs in AMD FPGAs, processing over 800 million bfloat16 numbers per second. This framework is then extended to variable precision, variable range, compressed numerical data types that are a user defined super set of both floats and posits [2]. The paper then discusses a simple hardware implementation of such format based on ANS (Asymmetrical Numeral Systems) [3] that acts as a bridge between this flexible data format and a computational engine while, at the same time, achieving bandwidth reduction. An example of a token factory using weight compression and sharing is also given.

Computer Vision and Pattern Recognition,Artificial Intelligence,Hardware Architecture
HybriDC: A Resource-Efficient CPU-FPGA Heterogeneous Acceleration System for Lossless Data Compression

Puguang Liu,Ziling Wei,Chuan Yu,Shuhui Chen

DOI: https://doi.org/10.3390/mi13112029

2022-11-19

Abstract:Lossless data compression is a crucial and computing-intensive application in data-centric scenarios. To reduce the CPU overhead, FPGA-based accelerators have been proposed to offload compression workloads. However, most existing schemes have the problem of an imbalanced resource utilization and a poor practicability. In this paper, we propose HybriDC, an adaptive resource-efficient CPU-FPGA heterogeneous acceleration system for lossless data compression. Leveraging complementary advantages of the heterogeneous architecture, HybriDC provides a universal end-to-end compression acceleration framework with application compatibility and performance scalability. To optimize the hardware compression kernel design, we build a performance-resource model of the compression algorithm taking into account the design goal, compression performance, available resources, etc. According to the deduced resource-balanced design principle, the compression algorithm parameters are fine-tuned, which reduces 32% of the block RAM usage of the LZ4 kernel. In the parallel compression kernel implementation, a memory-efficient parallel hash table with an extra checksum is proposed, which supports parallel processing and improves the compression ratio without extra memory. We develop an LZ4-based HybriDC system prototype and evaluate it in detail. Our LZ4 compression kernel achieves state-of-the-art memory efficiency, 2.5-4× better than existing designs with comparable compression ratios. The evaluation of total resource utilization and end-to-end throughput demonstrates the excellent scalability of HybriDC. In power efficiency, the four-kernel HybriDC prototype achieves a threefold advantage over the standard LZ4 algorithm.
An Efficient Lossless Compression Method for Internet Search Data in Hardware Accelerators

Jing Yan,Rong Luo,Rui Gao,Ning-Yi Xu

DOI: https://doi.org/10.1109/csie.2009.340

2009-01-01

Abstract:Nowadays machine learning algorithms are intensively used to improve the relevance of search engines by training on Internet search data, while their software implementations on commodity computers are not efficient (in terms of computation speed, power consumption, etc). Therefore FPGA-based accelerators have been proposed to process these large scale data. Data compression/decompression technology plays an important role for it could significantly increase those accelerators’ performance. Based on the statistics on datasets and the analysis of conventional data compression algorithms, we propose a bit mapping compression/decompression method to provide non-blocking streaming data to accelerators. Experiments indicate that the performance of hardware accelerator is increased up to 140% after adding the bit mapping modules. This method could also be extended to other data-intensive hardware accelerators.
A Compression Method for Inverted Index and Its FPGA-based Decompression Solution.

Jing Yan,Ning-Yi Xu,Zenglin Xia,Rong Luo,Feng-Hsiung Hsu

DOI: https://doi.org/10.1109/fpt.2010.5681488

2010-01-01

Abstract:Reconfigurable computing based on FPGA is a promising solution to accelerate applications for web search engines. Due to the challenge of such data-intensive applications, data compression has become much more important. This paper proposes a data compression method for inverted indices, which combines the bit-level compression method - Huffman coding and a coarse-grained compression method, to achieve a balanced performance in compression ratio and decompression speed. Because an inverted index is only compressed once, the compression speed is not the major measurement for a compression method. The proposed method shows good to 21.61% compression ratio on inverted indices from a commercial search engine. This compression ratio is better than results by other existing compression methods. We also develop an efficient FPGA-based hardware decompression module, which could provide up to 996 MBps input bandwidth for the accelerator system.

Fpga Implementation Of A Universal Lossless Data Compressor

Data Compression and Storage under High Speed Network

Multi-core Lossless Compression System for High-speed Data and Its FPGA Implementation

A Hardware Implementation of Real Time Lossless Data Compression and Decompression Circuits

FPGA bitstream compression and decompression using LZ and golomb coding (abstract only).

An Offloading Architecture of Lossless Compression Based on Smart NIC

Fpga Bitstream Compression and Decompression Based on Lz77 Algorithm and Bmc Technique

Method Of Lfm Pulse Compression Implementation Based On Fpga

UH-JLS: A Parallel Ultra-High Throughput JPEG-LS Encoding Architecture for Lossless Image Compression

An FPGA-Based LOCO-ANS Implementation for Lossless and Near-Lossless Image Compression Using High-Level Synthesis

FPGA Acceleration of Zstd Compression Algorithm

General Adaptive Lossless Compression for Multi-Channel Sensor Signals

Lossless Compression of Bitstream Configuration Files: Towards FPGA Cloud

Adaptive Pipeline Hardware Architecture Design and Implementation for Image Lossless Compression/Decompression Based on JPEG-LS

Lossless Compression Decoders for Bitstreams and Software Binaries Based on High-Level Synthesis.

Research and Implementation of FPGA Image Compression Parallel Algorithm Based on CCSDS

Design and Implementation of a Lossless Compression System for Hyperspectral Images

From a Lossless (~1.5:1) Compression Algorithm for Llama2 7B Weights to Variable Precision, Variable Range, Compressed Numeric Data Types for CNNs and LLMs

HybriDC: A Resource-Efficient CPU-FPGA Heterogeneous Acceleration System for Lossless Data Compression

An Efficient Lossless Compression Method for Internet Search Data in Hardware Accelerators

A Compression Method for Inverted Index and Its FPGA-based Decompression Solution.