Abstract:In the emerging edge-computing scenarios, FPGAs have been widely adopted to accelerate convolutional neural network (CNN)–based image-processing applications, such as image classification, object detection, and image segmentation, and so on. A standard image-processing pipeline first decodes the collected compressed images from Internet of Things (IoTs) to RGB data, then feeds them into CNN engines to compute the results. Previous works mainly focus on optimizing the CNN inference parts. However, we notice that on the popular ZYNQ FPGA platforms, image decoding can also become the bottleneck due to the poor performance of embedded ARM CPUs. Even with a hardware accelerator, the decoding operations still incur considerable latency. Moreover, conventional RGB-based CNNs have too few input channels at the first layer, which can hardly utilize the high parallelism of CNN engines and greatly slows down the network inference. To overcome these problems, in this article, we propose FD-CNN, a novel CNN accelerator leveraging the partial-decoding technique to accelerate CNNs directly in the frequency domain. Specifically, we omit the most time-consuming IDCT (Inverse Discrete Cosine Transform) operations of image decoding and directly feed the DCT coefficients (i.e., the frequency data) into CNNs. By this means, the image decoder can be greatly simplified. Moreover, compared to the RGB data, frequency data has a narrower input resolution but has 64× more channels. Such an input shape is more hardware friendly than RGB data and can substantially reduce the CNN inference time. We then systematically discuss the algorithm, architecture, and command set design of FD-CNN. To deal with the irregularity of different CNN applications, we propose an image-decoding-aware design-space exploration (DSE) workflow to optimize the pipeline. We further propose an early stopping strategy to tackle the time-consuming progressive JPEG decoding. Comprehensive experiments demonstrate that FD-CNN achieves, on average, 3.24×, 4.29× throughput improvement, 2.55×, 2.54× energy reduction and 2.38×, 2.58× lower latency on ZC-706 and ZCU-102 platforms, respectively, compared to the baseline image-processing pipelines.

Pure Frequency-Domain Deep Neural Network for IoT-enabled Smart Cameras

Alternative deep learning method for fast spatial-frequency shift imaging microscopy.

A Scalable Real-time Semantic Segmentation Network for Autonomous Driving

Real-time Semantic Segmentation with Weighted Factorized-Depthwise Convolution

On-chip 4F-System Based on Concave Mirrors for Optical Neural Networks

Adaptive Dila-DenseNet for Image Based Time Series Classification in IoT

Ultracompact Deep Neural Network for Ultrafast Optical Property Extraction in Spatial Frequency Domain Imaging (SFDI)

Video Scene Segmentation Using Tensor-Train Faster-RCNN for Multimedia IoT Systems.

STFNets: Learning Sensing Signals from the Time-Frequency Perspective with Short-Time Fourier Neural Networks

Deploy Large-Scale Deep Neural Networks in Resource Constrained IoT Devices with Local Quantization Region

Rotating Kernel CNN Optimization for Efficient IoT Surveillance on Low-Power Devices

An improved pulse coupled neural networks model for semantic IoT

Sophisticated Deep Learning with On-Chip Optical Diffractive Tensor Processing

Adaptive Network Configuration for Efficient and Accurate Neural Video Inference

Spectral Convolutional Neural Network Chip for In-sensor Edge Computing of Incoherent Natural Light

Fully Quantized Always-on Face Detector Considering Mobile Image Sensors

FastDeepIoT: Towards Understanding and Optimizing Neural Network Execution Time on Mobile and Embedded Devices

CS-CNN: Enabling Robust and Efficient Convolutional Neural Networks Inference for Internet-of-Things Applications

An Efficient Face Detector on a CPU Using Dual-Camera Sensors for Intelligent Surveillance Systems

Light-weight AI and IoT Collaboration for Surveillance Video Pre-Processing.

FD-CNN: A Frequency-Domain FPGA Acceleration Scheme for CNN-Based Image-Processing Applications