Abstract:Recurrent Neural Networks (RNNs) are becoming increasingly important for time series-related applications which require efficient and real-time implementations. The two major types are Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks. It is a challenging task to have real-time, efficient, and accurate hardware RNN implementations because of the high sensitivity to imprecision accumulation and the requirement of special activation function implementations. A key limitation of the prior works is the lack of a systematic design optimization framework of RNN model and hardware implementations, especially when the block size (or compression ratio) should be jointly optimized with RNN type, layer size, etc. In this paper, we adopt the block-circulant matrix-based framework, and present the Efficient RNN (E-RNN) framework for FPGA implementations of the Automatic Speech Recognition (ASR) application. The overall goal is to improve performance/energy efficiency under accuracy requirement. We use the alternating direction method of multipliers (ADMM) technique for more accurate block-circulant training, and present two design explorations providing guidance on block size and reducing RNN training trials. Based on the two observations, we decompose E-RNN in two phases: Phase I on determining RNN model to reduce computation and storage subject to accuracy requirement, and Phase II on hardware implementations given RNN model, including processing element design/optimization, quantization, activation implementation, etc. Experimental results on actual FPGA deployments show that E-RNN achieves a maximum energy efficiency improvement of 37.4$\times$ compared with ESE, and more than 2$\times$ compared with C-LSTM, under the same accuracy.

E-ERA: an Energy-Efficient Reconfigurable Architecture for RNNs Using Dynamically Adaptive Approximate Computing

EERA-ASR: an Energy-Efficient Reconfigurable Architecture for Automatic Speech Recognition with Hybrid DNN and Approximate Computing

EERA-DNN: an Energy-Efficient Reconfigurable Architecture for DNNs with Hybrid Bit-Width and Logarithmic Multiplier.

An Energy-efficient Reconfigurable Hybrid DNN Architecture for Speech Recognition with Approximate Computing.

ERA-LSTM: An Efficient ReRAM-Based Architecture for Long Short-Term Memory

E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAs

A 3.89-Gops/mw Scalable Recurrent Neural Network Processor with Improved Efficiency on Memory and Computation

AxR-NN: Approximate Computation Reuse for Energy-Efficient Convolutional Neural Networks

AERIS: Area/Energy-Efficient lT2R ReRAM Based Processing-in-Memory Neural Network System-on-a-Chip

E-PUR: An Energy-Efficient Processing Unit for Recurrent Neural Networks

An Energy-Efficient and Noise-Tolerant Recurrent Neural Network Using Stochastic Computing

Background Noise Adaptive Energy-Efficient Keywords Recognition Processor with Reusable DNN and Reconfigurable Architecture

Neural Approximating Architecture Targeting Multiple Application Domains

Reconfigurable Architecture for Neural Approximation in Multimedia Computing.

Energy-efficient Neural Networks Using Approximate Computation Reuse.

AERIS - area/energy-efficient 1T2R ReRAM based processing-in-memory neural network system-on-a-chip.

ARA: Cross-Layer Approximate Computing Framework Based Reconfigurable Architecture for CNNs

Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks

EERA-KWS: A 163 TOPS/W Always-on Keyword Spotting Accelerator in 28nm CMOS Using Binary Weight Network and Precision Self-Adaptive Approximate Computing.

RNA: Reconfigurable LSTM Accelerator with Near Data Approximate Processing

Digit-Serial DA-Based Fixed-Point RNNs: A Unified Approach for Enhancing Architectural Efficiency