Abstract:Training of convolutional neural networks (CNN) consumes a lot of time and resources. While most previous works have focused on accelerating the convolutional (CONV) layer, the proportion of non-convolutional (non-CONV) layers, such as batch normalization, is gradually increasing during training. Non-CONV layers have low cache reuse and arithmetic intensity, thereby performance is limited by memory bandwidth. Processing-in-memory (PIM) can utilize wide memory bandwidth, making it suitable for acceleration of non-CONV layers. Therefore, it makes sense to perform the computationally complex CONV layer on the host and handle the memory bottleneck challenges of the non-CONV layer on the PIM. Further improved performance can be expected if they run simultaneously. However, memory access conflicts between the host and PIM are the biggest factors hindering performance improvement. Prior studies proposed bank partitioning to alleviate memory conflicts, but it is not effective because CNN training involves significant data sharing between CONV and non-CONV layers. In this paper, we propose a memory scheduling and CNN training flow for the pipelined execution of CONV layers on the host and non-CONV layers on PIM. First, instead of applying bank partitioning, the host and PIM exclusively access memory for a certain period to avoid the movement of shared data between host memory and PIM memory. The conditions for switching the memory access authority between the host and PIM are set per layer, taking into account memory access characteristics and the number of queued memory requests. Second, in the training flow, CONV and non-CONV layers are pipelined in units of output feature map channels. Specifically, for the backward pass, the non-CONV tasks of the feature map gradient calculation phase and the weight gradient update phase are rearranged so that they can be easily performed within CONV layers. Experimental results show that the proposed pipelined execution achieves an average speedup of 18.1% at the network level compared to the serial operation of the host and PIM.

Accelerating CNNs Using Optimized Scheduling Strategy.

Performance Analysis of Different Convolution Algorithms in GPU Environment

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

A GPU-based high-performance optimization method of sparse convolutional neural networks

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Accelerating convolutional neural network by exploiting sparsity on GPUs

Optimizing CNN Hardware Acceleration with Configurable Vector Units and Feature Layout Strategies

Convolution Without Multiplication: A General Speed Up Strategy for CNNs

Optimizing of Convolutional Neural Network Accelerator

Optimizing Convolutional Neural Networks on Multi-Core Vector Accelerator

Recent Advances in Convolutional Neural Network Acceleration

A 681 GOPS/W~3.59 TOPS/W CNN Accelerator Based on Novel Data Flow Scheduling Scheme

CPU-Accelerator Co-Scheduling for CNN Acceleration at the Edge

TurboDL: Improving the CNN Training on GPU with Fine-Grained Multi-Streaming Scheduling

A High Efficient Architecture for Convolution Neural Network Accelerator

Multiple CNN-based Tasks Scheduling Across Shared GPU Platform in Research and Development Scenarios.

A Parallel Loading Based Accelerator for Convolution Neural Network

Accelerating CNN Training With Concurrent Execution of GPU and Processing-in-Memory

Optimizing the Convolution Operation to Accelerate Deep Neural Networks on FPGA

Optimizing Loop Operation and Dataflow in Fpga Acceleration of Deep Convolutional Neural Networks