Abstract:Training of convolutional neural networks (CNN) consumes a lot of time and resources. While most previous works have focused on accelerating the convolutional (CONV) layer, the proportion of non-convolutional (non-CONV) layers, such as batch normalization, is gradually increasing during training. Non-CONV layers have low cache reuse and arithmetic intensity, thereby performance is limited by memory bandwidth. Processing-in-memory (PIM) can utilize wide memory bandwidth, making it suitable for acceleration of non-CONV layers. Therefore, it makes sense to perform the computationally complex CONV layer on the host and handle the memory bottleneck challenges of the non-CONV layer on the PIM. Further improved performance can be expected if they run simultaneously. However, memory access conflicts between the host and PIM are the biggest factors hindering performance improvement. Prior studies proposed bank partitioning to alleviate memory conflicts, but it is not effective because CNN training involves significant data sharing between CONV and non-CONV layers. In this paper, we propose a memory scheduling and CNN training flow for the pipelined execution of CONV layers on the host and non-CONV layers on PIM. First, instead of applying bank partitioning, the host and PIM exclusively access memory for a certain period to avoid the movement of shared data between host memory and PIM memory. The conditions for switching the memory access authority between the host and PIM are set per layer, taking into account memory access characteristics and the number of queued memory requests. Second, in the training flow, CONV and non-CONV layers are pipelined in units of output feature map channels. Specifically, for the backward pass, the non-CONV tasks of the feature map gradient calculation phase and the weight gradient update phase are rearranged so that they can be easily performed within CONV layers. Experimental results show that the proposed pipelined execution achieves an average speedup of 18.1% at the network level compared to the serial operation of the host and PIM.

Accelerated Local Training of CNNs by Optimized Direct Feedback Alignment Based on Stochasticity of 4 Mb C-doped Ge<sub>2</sub>Sb<sub>2</sub>Te<sub>5</sub> PCM Chip in 40 nm Node

BOOST: Block Minifloat-Based On-Device CNN Training Accelerator with Transfer Learning

DaDianNao: A Machine-Learning Supercomputer

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks

An Efficient Learning Algorithm for Direct Training Deep Spiking Neural Networks

Direct Feedback Alignment with Sparse Connections for Local Learning

Training Low Bitwidth Convolutional Neural Network on RRAM

Silicon Photonic Architecture for Training Deep Neural Networks with Direct Feedback Alignment

Optimizing CNN Hardware Acceleration with Configurable Vector Units and Feature Layout Strategies

Efficient Training Convolutional Neural Networks on Edge Devices with Gradient-pruned Sign-symmetric Feedback Alignment

Layer-Wise Mixed-Modes CNN Processing Architecture With Double-Stationary Dataflow and Dimension-Reshape Strategy

Accelerating CNN Training With Concurrent Execution of GPU and Processing-in-Memory

An On-chip Layer-wise Training Method for RRAM Based Computing-in-memory Chips.

Mini-batch Serialization: CNN Training with Inter-layer Data Reuse

Hardware accelerator for an accurate local stereo matching algorithm using binary neural network

An Efficient Neuromorphic Implementation of Temporal Coding-Based On-Chip STDP Learning

Accelerating Low Bit-Width Deep Convolution Neural Network in MRAM.

Hybrid Stochastic-Binary Computing for Low-Latency and High-Precision Inference of CNNs

ITT-RNA: Imperfection Tolerable Training for RRAM-Crossbar-Based Deep Neural-Network Accelerator