Abstract:Deep learning (DL) is becoming the cornerstone of numerous applications both in datacenters and at the edge. Specialized hardware is often necessary to meet the performance requirements of state-of-the-art DL models, but the rapid pace of change in DL models and the wide variety of systems integrating DL make it impossible to create custom computer chips for all but the largest markets. Field-programmable gate arrays (FPGAs) present a unique blend of reprogrammability and direct hardware execution that make them suitable for accelerating DL inference. They offer the ability to customize processing pipelines and memory hierarchies to achieve lower latency and higher energy efficiency compared to general-purpose CPUs and GPUs, at a fraction of the development time and cost of custom chips. Their diverse high-speed IOs also enable directly interfacing the FPGA to the network and/or a variety of external sensors, making them suitable for both datacenter and edge use cases. As DL has become an ever more important workload, FPGA architectures are evolving to enable higher DL performance. In this article, we survey both academic and industrial FPGA architecture enhancements for DL. First, we give a brief introduction on the basics of FPGA architecture and how its components lead to strengths and weaknesses for DL applications. Next, we discuss different styles of DL inference accelerators on FPGA, ranging from model-specific dataflow styles to software-programmable overlay styles. We survey DL-specific enhancements to traditional FPGA building blocks such as logic blocks, arithmetic circuitry, and on-chip memories, as well as new in-fabric DL-specialized blocks for accelerating tensor computations. Finally, we discuss hybrid devices that combine processors and coarse-grained accelerator blocks with FPGA-like interconnect and networks-on-chip, and highlight promising future research directions.

A survey of FPGA design for AI era

A Near Memory Computing FPGA Architecture for Neural Network Acceleration

FPGA based cellular neural network optimization: from design space to system.

[DL] A Survey of FPGA-based Neural Network Inference Accelerators

A Survey of FPGA-Based Neural Network Accelerator

New paradigm of FPGA-based computational intelligence from surveying the implementation of DNN accelerators

A General Neural Network Hardware Architecture on FPGA

Field-Programmable Gate Array Architecture for Deep Learning: Survey & Future Directions

High Performance Scalable FPGA Accelerator for Deep Neural Networks

A Survey of FPGA Based Deep Learning Accelerators: Challenges and Opportunities

FPGA: The super chip in the age of artificial intelligence

FPGA/DNN Co-Design: An Efficient Design Methodology for IoT Intelligence on the Edge

Adaptive design and implementation of automatic modulation recognition accelerator

An All-Digital Compute-In-Memory FPGA Architecture for Deep Learning Acceleration

Revealing Untapped DSP Optimization Potentials for FPGA-Based Systolic Matrix Engines

A Survey of Accelerator Architectures for Deep Neural Networks

Optimizing Neural Networks for Efficient FPGA Implementation: A Survey

HAO: Hardware-aware neural Architecture Optimization for Efficient Inference

WPU: A FPGA-based Scalable, Efficient and Software/Hardware Co-design Deep Neural Network Inference Acceleration Processor

An Overview of FPGA Based Deep Learning Accelerators: Challenges and Opportunities.

Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks