Abstract:For processing streaming events from a Dynamic Vision Sensor camera, two types of neural networks can be considered. One are spiking neural networks, where simple spike-based computation is suitable for low-power consumption, but the discontinuity in spikes can make the training complicated in terms of hardware. The other one are digital Complementary Metal Oxide Semiconductor (CMOS)-based neural networks that can be trained directly using the normal backpropagation algorithm. However, the hardware and energy overhead can be significantly large, because all streaming events must be accumulated and converted into histogram data, which requires a large amount of memory such as SRAM. In this paper, to combine the spike-based operation with the normal backpropagation algorithm, memristor–CMOS hybrid circuits are proposed for implementing event-driven neural networks in hardware. The proposed hybrid circuits are composed of input neurons, synaptic crossbars, hidden/output neurons, and a neural network's controller. Firstly, the input neurons perform preprocessing for the DVS camera's events. The events are converted to histogram data using very simple memristor-based latches in the input neurons. After preprocessing the events, the converted histogram data are delivered to an ANN implemented using synaptic memristor crossbars. The memristor crossbars can perform low-power Multiply–Accumulate (MAC) calculations according to the memristor's current–voltage relationship. The hidden and output neurons can convert the crossbar's column currents to the output voltages according to the Rectified Linear Unit (ReLU) activation function. The neural network's controller adjusts the MAC calculation frequency according to the workload of the event computation. Moreover, the controller can disable the MAC calculation clock automatically to minimize unnecessary power consumption. The proposed hybrid circuits have been verified by circuit simulation for several event-based datasets such as POKER-DVS and MNIST-DVS. The circuit simulation results indicate that the neural network's performance proposed in this paper is degraded by as low as 0.5% while saving as much as 79% in power consumption for POKER-DVS. The recognition rate of the proposed scheme is lower by 0.75% compared to the conventional one, for the MNIST-DVS dataset. In spite of this little loss, the power consumption can be reduced by as much as 75% for the proposed scheme.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is: when processing streaming events from Dynamic Vision Sensor (DVS) cameras, how to combine the low - power consumption advantage of Spiking Neural Networks (SNNs) with the direct back - propagation training advantage of traditional Complementary Metal Oxide Semiconductor (CMOS) - based Artificial Neural Networks (ANNs), while reducing hardware and energy costs. Specifically, the paper proposes a method of implementing event - driven neural networks using memristor - CMOS hybrid circuits, aiming to overcome the following challenges: 1. **Difficulty in training SNNs**: Due to the discontinuity of spike signals in SNNs, it is very difficult to directly use traditional back - propagation methods for training. Although some methods such as surrogate gradient learning can alleviate this problem, these methods are still complex and energy - consuming in hardware implementation. 2. **High energy consumption of traditional digital CMOS neural networks**: Although traditional digital CMOS neural networks can be directly trained using conventional back - propagation algorithms, a large amount of memory (such as SRAM) is required to store the accumulated event data when processing streaming events, which leads to significant hardware and energy costs. To solve the above problems, the paper proposes a memristor - CMOS hybrid circuit that can efficiently process the streaming events generated by DVS cameras and achieve low - power consumption and efficient computing in the following ways: - **Input neurons**: Convert the spike events of the DVS camera into histogram data and use a simple memristor - based latch to complete pre - processing. - **Synaptic memristor cross - bar array**: Use the current - voltage relationship of memristors to perform low - power Multiply - Accumulate (MAC) calculations. - **Hidden / output neurons**: Convert the column current of the cross - bar array into an output voltage and use the ReLU activation function. - **Controller**: Adjust the MAC calculation frequency according to the workload of event processing and automatically disable the MAC calculation clock when necessary to reduce unnecessary power consumption. Through these designs, the hybrid circuit proposed in the paper shows good performance and significant energy - saving effects when processing streaming events. For example, for the POKER - DVS dataset, the performance degradation is no more than 0.5% while the power consumption savings are as high as 79%; for the MNIST - DVS dataset, the recognition rate only drops by 0.75% but the power consumption savings can reach 75%.

Memristor–CMOS Hybrid Circuits Implementing Event-Driven Neural Networks for Dynamic Vision Sensor Camera

Hybrid Dual-Complementary Metal-Oxide-semiconductor/memristor Synapse-Based Neural Network with Its Applications in Image Super-Resolution

Dynamic neural network with memristive CIM and CAM for 2D and 3D vision

Semantic memory-based dynamic neural network using memristive ternary CIM and CAM for 2D and 3D vision

Neural Networks Based on In-Sensor Computing of Optoelectronic Memristor

Efficient In Situ Learning of Hybrid LIF Neurons Using WTA Mechanism for High-Speed Low-Power Neuromorphic Systems

Memristor-based Deep Spiking Neural Network with a Computing-In-Memory Architecture

Fast and Accurate Sparse Coding of Visual Stimuli With a Simple, Ultralow-Energy Spiking Architecture

A Memristor-Based Processing-in-Memory Architecture for Deep Convolutional Neural Networks Approximate Computation

Memristor‐Based Analog Computation and Neural Network Classification with a Dot Product Engine

Memristor-Based Cellular Nonlinear/Neural Network: Design, Analysis, and Applications

Dynamics of Dual Memristors-based Neuron Circuit for Pattern Recognition

Training Itself: Mixed-signal Training Acceleration for Memristor-Based Neural Network

In-situ Learning in Hardware Compatible Multi-layer Memristive Spiking Neural Network

Hybrid CMOS-Memristor synapse circuits for implementing Ca ion-based plasticity model

Design of Artificial Neuron Network with Synapse Utilizing Hybrid CMOS Transistors with Memristor for Low Power Applications

Fully hardware-implemented memristor convolutional neural network

DS-CIM: A 40nm Asynchronous Dual-Spike Driven, MRAM Compute-In-Memory Macro for Spiking Neural Network

A Low-Cost Hardware-Friendly Spiking Neural Network Based on Binary MRAM Synapses, Accelerated Using In-Memory Computing

Spatial-Temporal Hybrid Neural Network With Computing-in-Memory Architecture

Adaptive Sparse Coding Based on Memristive Neural Network with Applications