Abstract:Training of convolutional neural networks (CNN) consumes a lot of time and resources. While most previous works have focused on accelerating the convolutional (CONV) layer, the proportion of non-convolutional (non-CONV) layers, such as batch normalization, is gradually increasing during training. Non-CONV layers have low cache reuse and arithmetic intensity, thereby performance is limited by memory bandwidth. Processing-in-memory (PIM) can utilize wide memory bandwidth, making it suitable for acceleration of non-CONV layers. Therefore, it makes sense to perform the computationally complex CONV layer on the host and handle the memory bottleneck challenges of the non-CONV layer on the PIM. Further improved performance can be expected if they run simultaneously. However, memory access conflicts between the host and PIM are the biggest factors hindering performance improvement. Prior studies proposed bank partitioning to alleviate memory conflicts, but it is not effective because CNN training involves significant data sharing between CONV and non-CONV layers. In this paper, we propose a memory scheduling and CNN training flow for the pipelined execution of CONV layers on the host and non-CONV layers on PIM. First, instead of applying bank partitioning, the host and PIM exclusively access memory for a certain period to avoid the movement of shared data between host memory and PIM memory. The conditions for switching the memory access authority between the host and PIM are set per layer, taking into account memory access characteristics and the number of queued memory requests. Second, in the training flow, CONV and non-CONV layers are pipelined in units of output feature map channels. Specifically, for the backward pass, the non-CONV tasks of the feature map gradient calculation phase and the weight gradient update phase are rearranged so that they can be easily performed within CONV layers. Experimental results show that the proposed pipelined execution achieves an average speedup of 18.1% at the network level compared to the serial operation of the host and PIM.

Mini-batch Serialization: CNN Training with Inter-layer Data Reuse

EWS: an Energy-Efficient CNN Accelerator with Enhanced Weight Stationary Dataflow

DaDianNao: A Machine-Learning Supercomputer

Communication Minimized Model-Architecture Co-design for Efficient Convolution Acceleration

Memory optimization of CNN Heterogeneous Multi-Core Architecture

Restructuring Batch Normalization to Accelerate CNN Training

LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reduction

Accelerating CNN Training With Concurrent Execution of GPU and Processing-in-Memory

Layup: Layer-adaptive and Multi-type Intermediate-oriented Memory Optimization for GPU-based CNNs

Bandwidth Efficient Architectures for Convolutional Neural Network

Reliable and Energy Efficient MLC STT-RAM Buffer for CNN Accelerators

Reducing SRAM Reading Power with Column Data Segment and Weights Correlation Enhancement for CNN Processing.

High Throughput CNN Inference and Training with In-Cache Computation

A Balanced Sparse Matrix Convolution Accelerator for Efficient CNN Training

UCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition

Accelerating Low Bit-Width Deep Convolution Neural Network in MRAM.

An Efficient Streaming Accelerator for Low Bit-Width Convolutional Neural Networks

CNN-DMA: A Predictable and Scalable Direct Memory Access Engine for Convolutional Neural Network with Sliding-window Filtering.

CMDS: Cross-layer Dataflow Optimization for DNN Accelerators Exploiting Multi-bank Memories

A High Efficient Architecture for Convolution Neural Network Accelerator

Optimally Scheduling CNN Convolutions for Efficient Memory Access