Abstract:Memristive neural network (MNN) has emerged as a new computing architecture with high speed and low power consumption, but its hardware implementation is hampered mainly by the device non-idealities of memristors. Here, we propose a backpropagation with gradient accumulation (BP-GA) algorithm which can effectively tolerate the memristor non-idealities. We first show that a memristor-based single-layer perceptron trained with BP-GA achieves high image recognition accuracies (>85% on the MNIST dataset) with a wide range of learning rates (0.01-50), large nonlinearities (>6), a small number of conductance states (even down to 3 states), a broad spectrum of ON/OFF ratios (across 3 orders of magnitude), and large noises (up to 40%). The origin for such good robustness against the learning rate and memristor non-idealities is then investigated and revealed to be associated with the operation mechanism of BP-GA. In this algorithm, the weights can keep increasing in magnitude due to the gradient accumulation and therefore become movable even at a small learning rate (or a large ON/OFF ratio equivalently). Moreover, the weights corresponding to the foreground and background of the input image are appropriately moved to the positive and negative boundaries of the weight range, respectively, thus allowing the learning of the image features using only a small number of conductance states. These boundary weights are trapped there without significant oscillation under the effects of accumulated gradients, resulting in immunity to large nonlinearity and noise. Therefore, BP-GA enables the MNN to be insensitive to the learning rate and robust against the memristor non-idealities. The performance of BP-GA is further evaluated on the memristor-based multilayer perceptron (on the MNIST dataset) and convolutional neural network (on the Cifar-10 dataset). For both MNNs, BP-GA exhibits relatively high accuracies and good tolerance against memristor non-idealities, demonstrating its applicability to complex networks and problems. This study provides a viable approach at the algorithm level for addressing some important hardware implementation issues of MNNs using realistic memristors.

An On-Chip BP Learning Neural Network with Ideal Neuron Characteristics and Learning Rate Adaptation

Hardware implementation of an on-chip BP learning neural network with programmable neuron characteristics and learning rate adaptation

A programmable on-chip BP learning neural network with enhanced neuron characteristics

Analogue Circuit Realization of a Programmable Sigmoidal Function and Its Derivative for On-Chip BP Learning

In-Hardware Learning of Multilayer Spiking Neural Networks on a Neuromorphic Processor

On-chip learning in a conventional silicon MOSFET based Analog Hardware Neural Network

CMOS Circuit Implementation of Spiking Neural Network for Pattern Recognition Using On-chip Unsupervised STDP Learning

A backpropagation with gradient accumulation algorithm capable of tolerating memristor non-idealities for training memristive neural networks

Hardware Realization of Building Blocks for Artificial Neural Networks

A Spike-Event-Based Neuromorphic Processor with Enhanced On-Chip STDP Learning in 28nm CMOS

Intralayer-Connected Spiking Neural Network with Hybrid Training Using Backpropagation and Probabilistic Spike-Timing Dependent Plasticity

A CMOS Spiking Neuron for Brain-Inspired Neural Networks with Resistive Synapses and In-Situ Learning

An Energy Efficient STDP-Based SNN Architecture With On-Chip Learning

A 4096-Neuron 1M-Synapse 3.8-pJ/SOP Spiking Neural Network With On-Chip STDP Learning and Sparse Weights in 10-nm FinFET CMOS

Training Multilayer Neural Network Based on Optimal Control Theory for Limited Computational Resources

CMOS Implementation of ANNs Based on Analog Optimization of N-Dimensional Objective Functions

In-situ Learning in Hardware Compatible Multi-layer Memristive Spiking Neural Network

H2Learn: High-Efficiency Learning Accelerator for High-Accuracy Spiking Neural Networks

Analog Spiking Neuron in CMOS 28 nm Towards Large-Scale Neuromorphic Processors

A Low-Power Spiking Neural Network Chip Based on a Compact LIF Neuron and Binary Exponential Charge Injector Synapse Circuits

A low cost neuromorphic learning engine based on a high performance supervised SNN learning algorithm