Abstract:Many long short-term memory (LSTM) applications need fast yet compact models. Neural network compression approaches, such as the grow-and-prune paradigm, have proved to be promising for cutting down network complexity by skipping insignificant weights. However, current compression strategies are mostly hardware-agnostic and network complexity reduction does not always translate into execution efficiency. In this work, we propose a hardware-guided symbiotic training methodology for compact, accurate, yet execution-efficient inference models. It is based on our observation that hardware may introduce substantial non-monotonic behavior, which we call the latency hysteresis effect, when evaluating network size vs. inference latency. This observation raises question about the mainstream smaller-dimension-is-better compression strategy, which often leads to a sub-optimal model architecture. By leveraging the hardware-impacted hysteresis effect and sparsity, we are able to achieve the symbiosis of model compactness and accuracy with execution efficiency, thus reducing LSTM latency while increasing its accuracy. We have evaluated our algorithms on language modeling and speech recognition applications. Relative to the traditional stacked LSTM architecture obtained for the Penn Treebank dataset, we reduce the number of parameters by 18.0x (30.5x) and measured run-time latency by up to 2.4x (5.2x) on Nvidia GPUs (Intel Xeon CPUs) without any accuracy degradation. For the DeepSpeech2 architecture obtained for the AN4 dataset, we reduce the number of parameters by 7.0x (19.4x), word error rate from 12.9% to 9.9% (10.4%), and measured run-time latency by up to 1.7x (2.4x) on Nvidia GPUs (Intel Xeon CPUs). Thus, our method yields compact, accurate, yet execution-efficient inference models.

Binarized LSTM Language Model.

Highly Efficient Neural Network Language Model Compression Using Soft Binarization Training

Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models

Structured Word Embedding For Low Memory Neural Network Language Model

PB-LLM: Partially Binarized Large Language Models

ARB-LLM: Alternating Refined Binarizations for Large Language Models

Neural Network Language Model Compression with Product Quantization and Soft Binarization

Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers

STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs

Major–Minor Long Short-Term Memory for Word-Level Language Model

FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation

Multi-View Lstm Language Model With Word-Synchronized Auxiliary Feature For Lvcsr

Compact Feedforward Sequential Memory Networks for Large Vocabulary Continuous Speech Recognition

Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM

BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance.

Compressed LSTM Using Balanced Sparsity

Future Vector Enhanced LSTM Language Model for LVCSR

Augmenting Language Models with Long-Term Memory

USM RNN-T model weights binarization

E-LSTM: an Efficient Hardware Architecture for Long Short-Term Memory

BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models