Abstract:Computing convolutional layers in the frequency domain using fast Fourier transformation (FFT) has been demonstrated to be effective in reducing the computational complexity of convolutional neural networks (CNNs). Nevertheless, the main challenge of this approach lies in the frequent and repeated transformations between the spatial and frequency domains due to the absence of nonlinear functions in the spectral domain, as such it makes the benefit less attractive for low-latency inference, especially on embedded platforms. To overcome the drawbacks in the existing FFT-based convolution, we propose a fully spectral CNN using a novel spectral-domain adaptive rectified linear unit (ReLU) layer, which completely removes the compute-intensive transformations between the spatial and frequency domains within the network. The proposed fully spectral CNNs maintain the nonlinearity of the spatial CNNs while taking into account the hardware efficiency. We then propose a deeply customized and compute-efficient hardware architecture to accelerate the fully spectral CNN inference on field programmable gate array (FPGA). Different hardware optimizations, such as spectral-domain intralayer and interlayer pipeline techniques, are introduced to further improve the performance of throughput. To achieve a load-balanced pipeline, a design space exploration (DSE) framework is proposed to optimize the resource allocation between hardware modules according to the resource constraints. On an Intel's Arria 10 SX160 FPGA, our optimized accelerator achieves a throughput of 204 Gop/s with 80% of compute efficiency. Compared with the state-of-the-art spatial and FFT-based implementations on the same device, our accelerator is 4× ∼ 6.6× and 3.0× ∼ 4.4× faster while maintaining a similar level of accuracy across different benchmark datasets.

An efficient implementation for linear convolution with reduced latency in FPGA

Optimizing Implementation of Gaussian Filter Based on Field Programmable Gate Array

An FPGA 2D-convolution unit based on the CAPH language

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

Efficient Inference of Large-Scale and Lightweight Convolutional Neural Networks on FPGA

Hardware Efficient Integrated In-loop Filter for HEVC Encoder

Implementation of In-Loop Filter for HEVC Decoder on Reconfigurable Processor

Efficient fast convolution architectures for convolutional neural network

Exploring Heterogeneous Algorithms for Accelerating Deep Convolutional Neural Networks on FPGAs

An FPGA-Based Accelerator Enabling Efficient Support for CNNs with Arbitrary Kernel Sizes

Design of Fully Spectral CNNs for Efficient FPGA-Based Acceleration

A Flexible and Efficient FPGA Accelerator for Various Large-Scale and Lightweight CNNs

Resource-Efficient Optimization for FPGA-Based Convolution Accelerator

Design of a Convolutional Two-Dimensional Filter in FPGA for Image Processing Applications

Configurable High Speed 2-D Convolution Processor and Gabor Filter Application

Low Area High-speed Hardware Implementation of Fast FIR Algorithm for Intelligent Signal Processing application in Complex Industrial Systems

Efficient Hardware Architectures for Deep Convolutional Neural Network

Hardware implementation of a series of transform matrices based on discrete hirschman transform

FPGA implementation of high performance digital FIR filter design using a hybrid adder and multiplier

An efficient VLSI architecture of 2‐D finite impulse response filter using enhanced approximate compressor circuits