Abstract:Recurrent neural networks (RNNs) have achieved the state-of-the-art performance on various sequence learning tasks due to their powerful sequence modeling capability. However, RNNs usually require a large number of parameters and high computational complexity. Hence, it is quite challenging to implement complex RNNs on embedded devices with stringent memory and latency requirement. In this paper, we first present a novel hybrid compression method for a widely used RNN variant, long-short term memory (LSTM), to tackle these implementation challenges. By properly using circulant matrices, forward nonlinear function approximation, and efficient quantization schemes with a retrain-based training strategy, the proposed compression method can reduce more than 95% of memory usage with negligible accuracy loss when verified under language modeling and speech recognition tasks. An efficient scalable parallel hardware architecture is then proposed for the compressed LSTM. With an innovative chessboard division method for matrix-vector multiplications, the parallelism of the proposed hardware architecture can be freely chosen under certain latency requirement. Specifically, for the circulant matrix-vector multiplications employed in the compressed LSTM, the circulant matrices are judiciously reorganized to fit in with the chessboard division and minimize the number of memory accesses required for the matrix multiplications. The proposed architecture is modeled using register transfer language (RTL) and synthesized under the TSMC 90-nm CMOS technology. With 518.5-kB on-chip memory, we are able to process a 512x512 compressed LSTM in 1.71 mu s, corresponding to 2.46 TOPS on the uncompressed one, at a cost of 30.77-mm(2) chip area. The implementation results demonstrate that the proposed design can achieve significantly high flexibility and area efficiency, which satisfies many real-time applications on embedded devices. It is worth mentioning that the memory-efficient approach of accelerating LSTM developed in this paper is also applicable to other RNN variants.

Run-Time Efficient RNN Compression for Inference on Edge Devices

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

Pushing the limits of RNN Compression

On-Demand Deep Model Compression for Mobile Devices

A 3.89-Gops/mw Scalable Recurrent Neural Network Processor with Improved Efficiency on Memory and Computation

Rank and run-time aware compression of NLP Applications

Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training

EdgeCompress: Coupling Multidimensional Model Compression and Dynamic Inference for EdgeAI

MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression

Accelerating Recurrent Neural Networks: A Memory-Efficient Approach

RDPNet: a Single-Path Lightweight CNN with Re-Parameterization for CPU-type Edge Devices

RNNAccel: A Fusion Recurrent Neural Network Accelerator for Edge Intelligence

Lightweight compression of neural network feature tensors for collaborative intelligence

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence

Compression of Recurrent Neural Networks using Matrix Factorization

To Compress, or Not to Compress: Characterizing Deep Learning Model Compression for Embedded Inference

Edge AI: Evaluation of Model Compression Techniques for Convolutional Neural Networks

Conv-inheritance: A hardware-efficient method to compress convolutional neural networks for edge applications

JMDC: A Joint Model and Data Compression System for Deep Neural Networks Collaborative Computing in Edge-Cloud Networks

Tensor train decompositions on recurrent networks