Abstract:ABSTRACTAccelerating the neural network inference by FPGA has emerged as a popular option, since the reconfigurability and high performance computing capability of FPGA intrinsically satisfies the computation demand of the fast-evolving neural algorithms. However, the popular neural accelerators on FPGA (e.g., Xilinx DPU) mainly utilize the DSP resources for constructing their processing units, while the rich LUT resources are not well exploited. Via the software-hardware co-design approach, in this work, we develop an FPGA-based heterogeneous computing system for neural network acceleration. From the hardware perspective, the proposed accelerator consists of DSP- and LUT-based GEneral Matrix-Multiplication (GEMM) computing cores, which forms the entire computing system in a heterogeneous fashion. The DSP- and LUT-based GEMM cores are computed w.r.t a unified Instruction Set Architecture (ISA) and unified buffers. Along the data flow of the neural network inference path, the computation of the convolution/fully-connected layer is split into two portions, handled by the DSP- and LUT-based GEMM cores asynchronously. From the software perspective, we mathematically and systematically model the latency and resource utilization of the proposed heterogeneous accelerator, regarding varying system design configurations. Through leveraging the reinforcement learning technique, we construct a framework to achieve end-to-end selection and optimization of the design specification of target heterogeneous accelerator, including workload split strategy, mixed-precision quantization scheme, and resource allocation of DSP- and LUT-core. In virtue of the proposed design framework and heterogeneous computing system, our design outperforms the state-of-the-art Mix&Match design with latency reduced by 1.12-1.32x with higher inference accuracy. The N3H-core is open-sourced at: https://github.com/elliothe/N3H_Core.

A Reconfigurable Parallel FPGA Accelerator for the Adapt-then-combine Diffusion LMS Algorithm

A Near Memory Computing FPGA Architecture for Neural Network Acceleration

A Ubiquitous Machine Learning Accelerator With Automatic Parallelization on FPGA

Design and implementation of LMS adaptive filter algorithm based on FPGA

A Reconfigurable Parallel FPGA Accelerator for the Kernel Affine Projection Algorithm

Design Exploration of Multi-FPGAs for Accelerating Deep Learning

Design and implementation of reconfigurable acceleration for in-memory distributed big data computing.

DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR

An SSD-MobileNet Acceleration Strategy for FPGAs Based on Network Compression and Subgraph Fusion

Exploring Heterogeneous Algorithms for Accelerating Deep Convolutional Neural Networks on FPGAs

An FPGA-Based Accelerator for Neighborhood-Based Collaborative Filtering Recommendation Algorithms

WooKong: A Ubiquitous Accelerator for Recommendation Algorithms with Custom Instruction Sets on FPGA

Efficient FPGA-Based Accelerator of the L-BFGS Algorithm for IoT Applications.

A FPGA Embedded DSP Supporting Parallel Multiple Low Bit-Width Multiply-Accumulate Operations

Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference

A flexible dataflow CNN accelerator on FPGA

N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores

Implementation and Optimization of the Accelerator Based on FPGA Hardware for LSTM Network

A 128-Way FPGA Platform for the Acceleration of KLMS Algorithm.

A Data-Center FPGA Acceleration Platform for Convolutional Neural Networks