Abstract:Hardware architectures composed of resistive cross-point device arrays can provide significant power and speed benefits for deep neural network training workloads using stochastic gradient descent (SGD) and backpropagation (BP) algorithm. The training accuracy on this imminent analog hardware however strongly depends on the switching characteristics of the cross-point elements. One of the key requirements is that these resistive devices must change conductance in a symmetrical fashion when subjected to positive or negative pulse stimuli. Here, we present a new training algorithm, so-called the "Tiki-Taka" algorithm, that eliminates this stringent symmetry requirement. We show that device asymmetry introduces an unintentional implicit cost term into the SGD algorithm, whereas in the "Tiki-Taka" algorithm a coupled dynamical system simultaneously minimizes the original objective function of the neural network and the unintentional cost term due to device asymmetry in a self-consistent fashion. We tested the validity of this new algorithm on a range of network architectures such as fully connected, convolutional and LSTM networks. Simulation results on these various networks show that whatever accuracy is achieved using the conventional SGD algorithm with symmetric (ideal) device switching characteristics the same accuracy is also achieved using the "Tiki-Taka" algorithm with non-symmetric (non-ideal) device switching characteristics. Moreover, all the operations performed on the arrays are still parallel and therefore the implementation cost of this new algorithm on array architectures is minimal; and it maintains the aforementioned power and speed benefits. These algorithmic improvements are crucial to relax the material specification and to realize technologically viable resistive crossbar arrays that outperform digital accelerators for similar training tasks.

Towards Exact Gradient-based Training on Analog In-memory Computing

Fast Algorithms for Exact IR Drop De-Embedding in Analog Multiply-Accumulate Computing

Pipeline Gradient-based Model Training on Analog In-memory Accelerators

Fast and robust analog in-memory deep neural network training

Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators

Towards training digitally-tied analog blocks via hybrid gradient computation

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Gradient descent-based programming of analog in-memory computing cores

Gradient Decomposition Methods for Training Neural Networks with Non-ideal Synaptic Devices

Fast offset corrected in-memory training

On the Accuracy of Analog Neural Network Inference Accelerators

Algorithm for Training Neural Networks on Resistive Device Arrays

Thermodynamic Natural Gradient Descent

Accuracy and Resiliency of Analog Compute-in-Memory Inference Engines

Low-Rank Gradient Descent for Memory-Efficient Training of Deep In-Memory Arrays

Towards Understanding the Generalizability of Delayed Stochastic Gradient Descent

Improving the Accuracy of Analog-Based In-Memory Computing Accelerators Post-Training

Asynchronous Accelerated Stochastic Gradient Descent.

Stepping Forward on the Last Mile

Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagation

Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends