Abstract:The training of neural network (NN) is usually time-consuming and resource intensive. Memristor has shown its potential in computation of NN. Especially for the metal-oxide resistive random access memory (RRAM), its crossbar structure and multi-bit characteristic can perform the matrix-vector product in high precision, which is the most common operation of NN. However, there exist two challenges on realizing the training of NN. Firstly, the current architecture can only support the inference phase of training and cannot perform the backpropagation (BP), the weights update of NN. Secondly, the training of NN requires enormous iterations and constantly updates the weights to reach the convergence, which leads to large energy consumption because of lots of write and read operations. In this work, we propose a novel architecture, TIME, and peripheral circuit designs to enable the training of NN in RRAM. TIME supports the BP and the weights update while maximizing the reuse of peripheral circuits for the inference operation on RRAM. Meanwhile, a variability-free tuning scheme and gradually-write circuits are designed to reduce the cost of tuning RRAM. We explore the performance of both SL (supervised learning) and DRL ( deep reinforcement learning) in TIME, and a specific mapping method of DRL is also introduced to further improve the energy efficiency. Experimental results show that, in SL, TIME can achieve 5.3x higher energy efficiency on average compared with the most powerful application-specific integrated circuits (ASIC) in the literature. In DRL, TIME can perform averagely 126x higher than GPU in energy efficiency. If the cost of tuning RRAM can be further reduced, TIME have the potential of boosting the energy efficiency by 2 orders of magnitude compared with ASIC.

In‐Memory Realization of Eligibility Traces Based on Conductance Drift of Phase Change Memory for Energy‐Efficient Reinforcement Learning

Energy-efficient Ferroelectric-FET-based Agent with Memory Trace for Enhanced Reinforcement Learning

Aging Aware Retraining for Memristor-based Neuromorphic Computing

Dual Memory Model for Experience-Once Task-Incremental Lifelong Learning.

Open-loop Analog Programmable Electrochemical Memory Array

PCM-trace: Scalable Synaptic Eligibility Traces with Resistivity Drift of Phase-Change Materials

Efficient Reinforcement Learning On Passive RRAM Crossbar Array

Phase-change Heterostructure Enables Ultralow Noise and Drift for Memory Operation.

In-memory and in-sensor reservoir computing with memristive devices

TIME: A Training-in-Memory Architecture for RRAM-Based Deep Neural Networks

Memristor Hardware-Friendly Reinforcement Learning

Sample Efficient Reinforcement Learning Using Graph-Based Memory Reconstruction.

Associative Memory Based Experience Replay for Deep Reinforcement Learning

SwiftRL: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systems

Reinforcement learning with analogue memristor arrays

Nonvolatile Electrochemical Random‐Access Memory under Short Circuit

Energy‐Efficient Memristive Euclidean Distance Engine for Brain‐Inspired Competitive Learning

Short-term Memory Traces for Action Bias in Human Reinforcement Learning

AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning

Battery life constrained real-time energy management strategy for hybrid electric vehicles based on reinforcement learning

Memristive Extreme Learning Machine: A Neuromorphic Implementation