Abstract:ABSTRACTAccelerating the neural network inference by FPGA has emerged as a popular option, since the reconfigurability and high performance computing capability of FPGA intrinsically satisfies the computation demand of the fast-evolving neural algorithms. However, the popular neural accelerators on FPGA (e.g., Xilinx DPU) mainly utilize the DSP resources for constructing their processing units, while the rich LUT resources are not well exploited. Via the software-hardware co-design approach, in this work, we develop an FPGA-based heterogeneous computing system for neural network acceleration. From the hardware perspective, the proposed accelerator consists of DSP- and LUT-based GEneral Matrix-Multiplication (GEMM) computing cores, which forms the entire computing system in a heterogeneous fashion. The DSP- and LUT-based GEMM cores are computed w.r.t a unified Instruction Set Architecture (ISA) and unified buffers. Along the data flow of the neural network inference path, the computation of the convolution/fully-connected layer is split into two portions, handled by the DSP- and LUT-based GEMM cores asynchronously. From the software perspective, we mathematically and systematically model the latency and resource utilization of the proposed heterogeneous accelerator, regarding varying system design configurations. Through leveraging the reinforcement learning technique, we construct a framework to achieve end-to-end selection and optimization of the design specification of target heterogeneous accelerator, including workload split strategy, mixed-precision quantization scheme, and resource allocation of DSP- and LUT-core. In virtue of the proposed design framework and heterogeneous computing system, our design outperforms the state-of-the-art Mix&Match design with latency reduced by 1.12-1.32x with higher inference accuracy. The N3H-core is open-sourced at: https://github.com/elliothe/N3H_Core.

Heterogeneous Systems with Reconfigurable Neuromorphic Computing Accelerators

A Near Memory Computing FPGA Architecture for Neural Network Acceleration

A heterogeneous computing system with memristor-based neuromorphic accelerators

N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores

UIC: A Unified and Scalable Chip Integrating Neuromorphic Computation and General Purpose Processor

Harmonica: A Framework of Heterogeneous Computing Systems with Memristor-Based Neuromorphic Computing Accelerators

Accelerating Hybrid And Compact Neural Networks Targeting Perception And Control Domains With Coarse-Grained Dataflow Reconfiguration

A Hybrid Heterogeneous Neural Network Accelerator Based on Systolic Array

N3H-Core

RENO: a high-efficient reconfigurable neuromorphic computing accelerator design

Back to Homogeneous Computing: A Tightly-Coupled Neuromorphic Processor With Neuromorphic ISA

Rna: A Reconfigurable Architecture for Hardware Neural Acceleration

NEUTRAMS: Neural Network Transformation and Co-Design under Neuromorphic Hardware Constraints

End-to-End Implementation of Various Hybrid Neural Networks on a Cross-Paradigm Neuromorphic Chip

A High Performance Reconfigurable Hardware Architecture for Lightweight Convolutional Neural Network

A High Energy-Efficiency Multi-core Neuromorphic Architecture for Deep SNN Training

MENAGE: Mixed-Signal Event-Driven Neuromorphic Accelerator for Edge Applications

A Heterogeneous Full-stack AI Platform for Performance Monitoring and Hardware-specific Optimizations

A Ubiquitous Machine Learning Accelerator With Automatic Parallelization on FPGA

An FPGA-based Accelerator for Deep Neural Network with Novel Reconfigurable Architecture.

A Versatile Neuromorphic System Based on Simple Neuron Model