Abstract:In our previous work we have shown that resistive cross point devices, so called Resistive Processing Unit (RPU) devices, can provide significant power and speed benefits when training deep fully connected networks as well as convolutional neural networks. In this work, we further extend the RPU concept for training recurrent neural networks (RNNs) namely LSTMs. We show that the mapping of recurrent layers is very similar to the mapping of fully connected layers and therefore the RPU concept can potentially provide large acceleration factors for RNNs as well. In addition, we study the effect of various device imperfections and system parameters on training performance. Symmetry of updates becomes even more crucial for RNNs; already a few percent asymmetry results in an increase in the test error compared to the ideal case trained with floating point numbers. Furthermore, the input signal resolution to device arrays needs to be at least 7 bits for successful training. However, we show that a stochastic rounding scheme can reduce the input signal resolution back to 5 bits. Further, we find that RPU device variations and hardware noise are enough to mitigate overfitting, so that there is less need for using dropout. We note that the models trained here are roughly 1500 times larger than the fully connected network trained on MNIST dataset in terms of the total number of multiplication and summation operations performed per epoch. Thus, here we attempt to study the validity of the RPU approach for large scale networks.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are: How to effectively train Recurrent Neural Networks (RNNs), especially Long Short - Term Memory networks (LSTMs), using Resistive Cross - Point Devices (RPU devices for short). Specifically, the authors hope to verify whether RPU devices can provide significant speed and power consumption advantages for the training of large - scale LSTM networks, and study the influence of device imperfections and system parameters on training performance. ### Summary of main problems: 1. **Application of RPU devices in LSTM training**: - Verify whether RPU devices can effectively accelerate the training of LSTM networks. - Evaluate the performance of RPU devices when dealing with large - scale LSTM networks. 2. **Influence of device imperfections and system parameters**: - Research the influence of factors such as device asymmetry, input signal resolution, and hardware noise on training accuracy. - Analyze whether these factors will lead to a decline in training performance and explore possible solutions. 3. **Effect of regularization techniques (such as Dropout)**: - Explore whether it is still necessary to use Dropout on RPU hardware, or whether hardware noise has already played a similar role. 4. **Applicability of large - scale networks**: - Test the performance of RPU devices when dealing with networks on a larger scale than the MNIST dataset, such as large - scale LSTM networks for character - level language models. ### Specific objectives: - **Map LSTM layers to RPU arrays**: Demonstrate how to map complex LSTM blocks to RPU arrays and test their training effects. - **Evaluate device characteristics**: Research the influence of different device characteristics and system parameters (such as update symmetry, input signal resolution, etc.) on training accuracy. - **Optimize training methods**: Explore training methods suitable for RPU devices, such as simple Stochastic Gradient Descent (SGD), and evaluate their effectiveness. - **Improve input signal resolution**: Propose the use of a random rounding scheme to improve input signal resolution, thereby improving training performance. Through these studies, the authors aim to prove the potential of RPU devices in training complex and large - scale LSTM networks and provide valuable references for future hardware design.

Training LSTM Networks with Resistive Cross-Point Devices

Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices

Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices

Analog CMOS-based Resistive Processing Unit for Deep Neural Network Training

Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training

Algorithm for Training Neural Networks on Resistive Device Arrays

E-PUR: An Energy-Efficient Processing Unit for Recurrent Neural Networks

System-level Impact of Non-Ideal Program-Time of Charge Trap Flash (CTF) on Deep Neural Network

Long Short-Term Memory Implementation Exploiting Passive RRAM Crossbar Array

Convolutional Neural Networks Based on RRAM Devices for Image Recognition and Online Learning Tasks

Low Bit-Width Convolutional Neural Network on RRAM

Training Low Bitwidth Convolutional Neural Network on RRAM

Training large-scale ANNs on simulated resistive crossbar arrays

Training Recurrent Neural Networks against Noisy Computations during Inference

Learning Device Models with Recurrent Neural Networks

TIME: A Training-in-Memory Architecture for RRAM-Based Deep Neural Networks

Unsupervised Learning on Resistive Memory Array Based Spiking Neural Networks

A Performance Review of Recurrent Neural Networks Long Short-Term Memory (LSTM)

Recurrent Neural Networks With Limited Numerical Precision

Spiking Neural Network with RRAM: Can We Use It for Real-World Application?

Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability.