Abstract:ABSTRACTAccelerating the neural network inference by FPGA has emerged as a popular option, since the reconfigurability and high performance computing capability of FPGA intrinsically satisfies the computation demand of the fast-evolving neural algorithms. However, the popular neural accelerators on FPGA (e.g., Xilinx DPU) mainly utilize the DSP resources for constructing their processing units, while the rich LUT resources are not well exploited. Via the software-hardware co-design approach, in this work, we develop an FPGA-based heterogeneous computing system for neural network acceleration. From the hardware perspective, the proposed accelerator consists of DSP- and LUT-based GEneral Matrix-Multiplication (GEMM) computing cores, which forms the entire computing system in a heterogeneous fashion. The DSP- and LUT-based GEMM cores are computed w.r.t a unified Instruction Set Architecture (ISA) and unified buffers. Along the data flow of the neural network inference path, the computation of the convolution/fully-connected layer is split into two portions, handled by the DSP- and LUT-based GEMM cores asynchronously. From the software perspective, we mathematically and systematically model the latency and resource utilization of the proposed heterogeneous accelerator, regarding varying system design configurations. Through leveraging the reinforcement learning technique, we construct a framework to achieve end-to-end selection and optimization of the design specification of target heterogeneous accelerator, including workload split strategy, mixed-precision quantization scheme, and resource allocation of DSP- and LUT-core. In virtue of the proposed design framework and heterogeneous computing system, our design outperforms the state-of-the-art Mix&Match design with latency reduced by 1.12-1.32x with higher inference accuracy. The N3H-core is open-sourced at: https://github.com/elliothe/N3H_Core.

MuGRA: A Scalable Multi-Grained Reconfigurable Accelerator Powered by Elastic Neural Network

A Near Memory Computing FPGA Architecture for Neural Network Acceleration

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

Multi-clusters: an Efficient Design Paradigm of NN Accelerator Architecture Based on FPGA

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

A Fine-Grained Sparse Accelerator for Multi-Precision DNN.

Heterogeneous Systems with Reconfigurable Neuromorphic Computing Accelerators

Mixed-granularity Parallel Coarse-Grained Reconfigurable Architecture

A Coarse-Grained Reconfigurable Architecture for Compute-Intensive MapReduce Acceleration

N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores

N3H-Core

DGNN-Booster: A Generic FPGA Accelerator Framework For Dynamic Graph Neural Network Inference

TuNao: A High-Performance and Energy-Efficient Reconfigurable Accelerator for Graph Processing

Reconfigurable Architecture for Neural Approximation in Multimedia Computing.

A High Performance Reconfigurable Hardware Architecture for Lightweight Convolutional Neural Network

A Reconfigurable Accelerator for Sparse Convolutional Neural Networks.

A Ubiquitous Machine Learning Accelerator With Automatic Parallelization on FPGA

Rna: A Reconfigurable Architecture for Hardware Neural Acceleration

Reconfigurable Binary Neural Network Accelerator with Adaptive Parallelism Scheme

MARCA: Mamba Accelerator with ReConfigurable Architecture

High-Performance FPGA-Based CNN Accelerator with Block-Floating-Point Arithmetic.