Abstract:Deep convolutional neural networks have achieved remarkable progress in recent years. However, the large volume of intermediate results generated during inference poses a significant challenge to the accelerator design for resource-constrained field-programmable gate array (FPGA). Due to the limited on-chip storage, partial results of intermediate layers are frequently transferred back and forth between on-chip memory and off-chip DRAM, leading to a nonnegligible increase in latency and energy consumption. In this article, we propose block convolution, a hardware-friendly, simple, yet efficient convolution operation that can completely avoid the off-chip transfer of intermediate feature maps at runtime. The fundamental idea of block convolution is to eliminate the dependency of feature map tiles in the spatial dimension when spatial tiling is used, which is realized by splitting a feature map into independent blocks so that convolution can be performed separately on individual blocks. We conduct extensive experiments to demonstrate the efficacy of the proposed block convolution on both the algorithm side and the hardware side. Specifically, we evaluate block convolution on: 1) VGG-16, ResNet-18, ResNet-50, and MobileNet-V1 for the ImageNet classification task; 2) SSD and FPN for the COCO object detection task; and 3) VDSR for the Set5 single-image superresolution task. Experimental results demonstrate that comparable or higher accuracy can be achieved with block convolution. We also showcase two CNN accelerators via algorithm/hardware co-design based on block convolution on memory-limited FPGAs, and evaluation shows that both accelerators substantially outperform the baseline without off-chip transfer of intermediate feature maps.

A Synthesis Flow For Fast Convolution Unit Based On Molecular Reactions

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

Efficient Convolution Architectures for Convolutional Neural Network

Convolution Without Multiplication: A General Speed Up Strategy for CNNs

F-DNA: Fast Convolution Architecture for Deconvolutional Network Acceleration

An Efficient Design Flow for Accelerating Complicated-connected CNNs on a Multi-FPGA Platform.

Efficient fast convolution architectures for convolutional neural network

A High Efficient Architecture for Convolution Neural Network Accelerator

Implementation of Convolutional Neural Network with Co-design of High-Level Synthesis and Verilog HDL

A High Performance FPGA-based Accelerator for Large-Scale Convolutional Neural Networks

Reconfigurable co-processor architecture with limited numerical precision to accelerate deep convolutional neural networks

DSCU: Accelerating CNN Inference in FPGAs with Dual Sizes of Compute Unit

Flexible and Efficient Convolutional Acceleration on Unified Hardware Using the Two-Stage Splitting Method and Layer-Adaptive Allocation of 1-D/2-D Winograd Units

A Reconfigurable Process Engine for Flexible Convolutional Neural Network Acceleration

Efficient Reconfigurable Hardware Core for Convolutional Neural Networks.

Efficient Hardware Architectures for Deep Convolutional Neural Network

A Reconfigurable Approach for Deconvolutional Network Acceleration with Fast Algorithm.

Frequency-Domain Inference Acceleration for Convolutional Neural Networks Using ReRAMs

Block Convolution: Towards Memory-Efficient Inference of Large-Scale CNNs on FPGA

Efficient Fast Convolution Architecture Based on Stochastic Computing