Abstract:Deep learning (DL) is becoming the cornerstone of numerous applications both in datacenters and at the edge. Specialized hardware is often necessary to meet the performance requirements of state-of-the-art DL models, but the rapid pace of change in DL models and the wide variety of systems integrating DL make it impossible to create custom computer chips for all but the largest markets. Field-programmable gate arrays (FPGAs) present a unique blend of reprogrammability and direct hardware execution that make them suitable for accelerating DL inference. They offer the ability to customize processing pipelines and memory hierarchies to achieve lower latency and higher energy efficiency compared to general-purpose CPUs and GPUs, at a fraction of the development time and cost of custom chips. Their diverse high-speed IOs also enable directly interfacing the FPGA to the network and/or a variety of external sensors, making them suitable for both datacenter and edge use cases. As DL has become an ever more important workload, FPGA architectures are evolving to enable higher DL performance. In this article, we survey both academic and industrial FPGA architecture enhancements for DL. First, we give a brief introduction on the basics of FPGA architecture and how its components lead to strengths and weaknesses for DL applications. Next, we discuss different styles of DL inference accelerators on FPGA, ranging from model-specific dataflow styles to software-programmable overlay styles. We survey DL-specific enhancements to traditional FPGA building blocks such as logic blocks, arithmetic circuitry, and on-chip memories, as well as new in-fabric DL-specialized blocks for accelerating tensor computations. Finally, we discuss hybrid devices that combine processors and coarse-grained accelerator blocks with FPGA-like interconnect and networks-on-chip, and highlight promising future research directions.

FPGA Oriented Lightweight Deep Learning Inference for Liver Cancer Segmentation

A Near Memory Computing FPGA Architecture for Neural Network Acceleration

Research on Lightweight Network Segmentation of Lung Parenchyma CT Image Based on FPGA Platform

FPGA based Flexible Implementation of Light Weight Inference on Deep Convolutional Neural Networks

FPGA Deployment of LFADS for Real-time Neuroscience Experiments

DCSegNet: Deep Learning Framework Based on Divide-and-Conquer Method for Liver Segmentation

Efficient Inference of Large-Scale and Lightweight Convolutional Neural Networks on FPGA

Deploying deep learning networks based advanced techniques for image processing on FPGA platform

FPGA Hardware Acceleration Design for Deep Learning

Lightweight skin cancer detection IP hardware implementation using cycle expansion and optimal computation arrays methods

An SSD-MobileNet Acceleration Strategy for FPGAs Based on Network Compression and Subgraph Fusion

MRI-based brain tumor segmentation using FPGA-accelerated neural network

Hardware Design of Lightweight Binary Classification Algorithms for Small-Size Images on FPGA

Enabling Efficient and Flexible FPGA Virtualization for Deep Learning in the Cloud

FPGA accelerates deep residual learning for image recognition

[DL] A Survey of FPGA-based Neural Network Inference Accelerators

Low-Cost FPGA Implementation of Deep Learning-Based Heart Sound Segmentation for Real-Time CVDs Screening

Compressing Recurrent Neural Networks for FPGA-accelerated Implementation in Fluorescence Lifetime Imaging

Field-Programmable Gate Array Architecture for Deep Learning: Survey & Future Directions

FPGA-Based In-Vivo Calcium Image Decoding for Closed-Loop Feedback Applications

Full-stack Optimization for Accelerating CNNs with FPGA Validation