Abstract:FPGAs provide a flexible and efficient platform to accelerate rapidly-changing algorithms for computer vision. The majority of existing work focuses on accelerating image classification, while other fundamental vision problems, including object detection and instance segmentation, have not been adequately addressed. Compared with image classification, detection problems are more sensitive to the spatial variance of objects, and therefore, require specialized convolutions to aggregate spatial information. To address this, recent work proposes dynamic deformable convolution to augment regular convolutions. Regular convolutions process a fixed grid of pixels across all the spatial locations in an image, while dynamic deformable convolutions may access arbitrary pixels in the image and the access pattern is input-dependent and varies per spatial location. These properties lead to inefficient memory accesses of inputs with existing hardware. In this work, we first investigate the overhead of the deformable convolution on embedded FPGA SoCs, and then show the accuracy-latency tradeoffs for a set of algorithm modifications including full versus depthwise, fixed-shape, and limited-range. These modifications benefit the energy efficiency for embedded devices in general as they reduce the compute complexity. We then build an efficient object detection network with modified deformable convolutions and quantize the network using state-of-the-art quantization methods. We implement a unified hardware engine on FPGA to support all the operations in the network. Preliminary experiments show that little accuracy is compromised and speedup can be achieved with our co-design optimization for the deformable convolution.

A Unified Acceleration Solution Based on Deformable Network for Image Pixel Processing

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

A Memory-Efficient Hardware Architecture for Deformable Convolutional Networks

Myocarditis: A clinical entity that can benefit from noninvasive imaging

Energy-Efficient Accelerator Design for Deformable Convolution Networks

A Unified Hardware Architecture for Convolutions and Deconvolutions in CNN

A High Performance Reconfigurable Hardware Architecture for Lightweight Convolutional Neural Network

Design of a Generic Dynamically Reconfigurable Convolutional Neural Network Accelerator with Optimal Balance

A FPGA-based Hardware Accelerator for Multiple Convolutional Neural Networks

Algorithm-hardware Co-design for Deformable Convolution

Design and Implementation of Convolutional Neural Network Image Recognition Acceleration System Based on FPGA

A High-Performance Pixel-Level Fully Pipelined Hardware Accelerator for Neural Networks

Towards a Uniform Architecture for the Efficient Implementation of 2D and 3D Deconvolutional Neural Networks on FPGAs

A Flexible and Efficient FPGA Accelerator for Various Large-Scale and Lightweight CNNs

Hardware Implementation of Depthwise Separable Convolution Neural Network

Efficient Hardware Architectures for Deep Convolutional Neural Network

DSA-CNN: an fpga-integrated deformable systolic array for convolutional neural network acceleration

An Efficient Accelerator for Multiple Convolutions From the Sparsity Perspective

A Design Methodology for Efficient Implementation of Deconvolutional Neural Networks on an FPGA

Communication-Aware and Resource-Efficient NoC-Based Architecture for CNN Acceleration