Abstract:Recurrent Neural Networks (RNNs) have become important tools for tasks such as speech recognition, text generation, or natural language processing. However, their inference may involve up to billions of operations and their large number of parameters leads to large storage size and runtime memory usage. These reasons impede the adoption of these models in real-time, on-the-edge applications. Field-Programmable Gate Arrays (FPGAs) and Application-Specific Integrated Circuits (ASICs) have emerged as promising solutions for the hardware acceleration of these algorithms, thanks to their degree of customization of compute data paths and memory subsystems, which makes them take the maximum advantage from compression techniques for what concerns area, timing, and power consumption. In contrast to the extensive study in compression and quantization for plain feed forward neural networks in the literature, little attention has been paid to reducing the computational resource requirements of RNNs. This work proposes a new effective methodology for the post-training quantization of RNNs. In particular, we focus on the quantization of Long Short-Term Memory (LSTM) RNNs and Gated Recurrent Unit (GRU) RNNs. The proposed quantization strategy is meant to be a detailed guideline toward the design of custom hardware accelerators for LSTM/GRU-based algorithms to be implemented on FPGA or ASIC devices using fixed-point arithmetic only. We applied our methods to LSTM/GRU models pretrained on the IMDb sentiment classification dataset and Penn TreeBank language modelling dataset, thus comparing each quantized model to its floating-point counterpart. The results show the possibility to achieve up to 90% memory footprint reduction in both cases, obtaining less than 1% loss in accuracy and even a slight improvement in the Perplexity per word metric, respectively. The results are presented showing the various trade-offs between memory footprint reduction and accuracy changes, demonstrating the benefits of the proposed methodology even in comparison with other works from the literature.

Fixed-point performance analysis of recurrent neural networks

Fixed-point optimization of deep neural networks with adaptive step size retraining

Resiliency of Deep Neural Networks under Quantization

Effective Quantization Methods for Recurrent Neural Networks

Fixed-Point Back-Propagation Training

Alternating Multi-bit Quantization for Recurrent Neural Networks

Recurrent Neural Networks With Limited Numerical Precision

Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm

A Post-training Quantization Method for the Design of Fixed-Point-Based FPGA/ASIC Hardware Accelerators for LSTM/GRU Algorithms

Learning Fixed Points of Recurrent Neural Networks by Reparameterizing the Network Model

Bit-shrinking: Limiting Instantaneous Sharpness for Improving Post-training Quantization

Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms

Fixed Point Quantization of Deep Convolutional Networks

Solving Oscillation Problem in Post-Training Quantization Through a Theoretical Perspective

Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients

Optimizing Quantized Neural Networks in a Weak Curvature Manifold

Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers

Unsupervised Network Quantization via Fixed-Point Factorization

On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic

Residual Quantization for Low Bit-Width Neural Networks.