Abstract:Computing convolutional layers in the frequency domain using fast Fourier transformation (FFT) has been demonstrated to be effective in reducing the computational complexity of convolutional neural networks (CNNs). Nevertheless, the main challenge of this approach lies in the frequent and repeated transformations between the spatial and frequency domains due to the absence of nonlinear functions in the spectral domain, as such it makes the benefit less attractive for low-latency inference, especially on embedded platforms. To overcome the drawbacks in the existing FFT-based convolution, we propose a fully spectral CNN using a novel spectral-domain adaptive rectified linear unit (ReLU) layer, which completely removes the compute-intensive transformations between the spatial and frequency domains within the network. The proposed fully spectral CNNs maintain the nonlinearity of the spatial CNNs while taking into account the hardware efficiency. We then propose a deeply customized and compute-efficient hardware architecture to accelerate the fully spectral CNN inference on field programmable gate array (FPGA). Different hardware optimizations, such as spectral-domain intralayer and interlayer pipeline techniques, are introduced to further improve the performance of throughput. To achieve a load-balanced pipeline, a design space exploration (DSE) framework is proposed to optimize the resource allocation between hardware modules according to the resource constraints. On an Intel's Arria 10 SX160 FPGA, our optimized accelerator achieves a throughput of 204 Gop/s with 80% of compute efficiency. Compared with the state-of-the-art spatial and FFT-based implementations on the same device, our accelerator is 4× ∼ 6.6× and 3.0× ∼ 4.4× faster while maintaining a similar level of accuracy across different benchmark datasets.

Applications of convolutional neural networks for spectral analysis

Spectral Imaging with Deep Learning.

Convolutional Neural Network Based Computational Imaging Spectroscopy

Spectral Representations for Convolutional Neural Networks

Advances in Convolutional Neural Networks

Principles and applications of convolutional neural network for spectral analysis in food quality evaluation: A review

Understanding the Learning Mechanism of Convolutional Neural Networks in Spectral Analysis

Convolutional neural networks for classification and regression analysis of one-dimensional spectral data

Convolutional neural nets in chemical engineering: Foundations, computations, and applications

A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects

Neural Networks: Computational Models and Applications - ReadingSample

Spectral Neural Networks: Approximation Theory and Optimization Landscape

The application of convolutional neural networks for tomographic reconstruction of hyperspectral images

High Efficient Deep Feature Extraction and Classification of Spectral-Spatial Hyperspectral Image Using Cross Domain Convolutional Neural Networks

Deep learning in spectral analysis: Modeling and imaging

A review of convolutional neural networks in computer vision

Understanding Spectral Graph Neural Network

Multicomponent Raman Spectral Regression Using Complete and Incomplete Models and Convolutional Neural Networks.

Convolutional networks and applications in vision

Design of Fully Spectral CNNs for Efficient FPGA-Based Acceleration