Abstract:Deep convolutional neural networks (CNNs) have achieved remarkable performance at the cost of huge computation. As the CNN models become more complex and deeper, compressing CNNs to sparse by pruning the redundant connection in the networks has emerged as an attractive approach to reduce the amount of computation and memory requirement. On the other hand, FPGAs have been demonstrated to be an effective hardware platform to accelerate CNN inference. However, most existing FPGA accelerators focus on dense CNN models, which are inefficient when executing sparse models as most of the arithmetic operations involve addition and multiplication with zero operands. In this work, we propose an accelerator with software-hardware co-design for sparse CNNs on FPGAs. To efficiently deal with the irregular connections in the sparse convolutional layers, we propose a weight-oriented dataflow that exploits element-matrix multiplication as the key operation. Each weight is processed individually, which yields low decoding overhead. Then, we design an FPGA accelerator that features a tile look-up table (TLUT) and a channel multiplexer (CMUX). The TLUT is designed to match the index between sparse weights and input pixels. Using TLUT, the runtime decoding overhead is mitigated by using an efficient indexing operation. Moreover, we propose a weight layout to enable efficient on-chip memory access without conflicts. To cooperate with the weight layout, a CMUX is inserted to locate the address. Finally, we build a neural architecture search (NAS) engine that leverages the reconfigurability of FPGAs to generate an efficient CNN model and choose the optimal hardware design parameters. The experiments demonstrate that our accelerator can achieve 223.4-309.0 GOP/s for the modern CNNs on Xilinx ZCU102, which provides a 2.4x-12.9x speedup over previous dense CNN accelerators on FPGAs. Our FPGA-aware NAS approach shows 2x speedup over MobileNetV2 with 1.5% accuracy loss.

An Efficient Channel-Aware Sparse Binarized Neural Networks Inference Accelerator

A High-Efficiency FPGA-Based Accelerator for Binarized Neural Network

FBNA: A Fully Binarized Neural Network Accelerator.

FCA-BNN: Flexible and Configurable Accelerator for Binarized Neural Networks on FPGA.

Sparsity-Aware Optimization of In-Memory Bayesian Binary Neural Network Accelerators

An Approach of Binary Neural Network Energy-Efficient Implementation

A Low-Power Sparse Convolutional Neural Network Accelerator with Pre-Encoding Radix-4 Booth Multiplier

Accelerating Binarized Convolutional Neural Networks with Software-Programmable FPGAs.

An Energy-Efficient Spiking Neural Network Accelerator Based on Spatio-Temporal Redundancy Reduction

TCP-Net: Minimizing Operation Counts of Binarized Neural Network Inference.

BinSparX: Sparsified Binary Neural Networks for Reduced Hardware Non-Idealities in Xbar Arrays

A high-throughput scalable BNN accelerator with fully pipelined architecture

A Fully Onchip Binarized Convolutional Neural Network FPGA Impelmentation with Accurate Inference.

Reconfigurable Binary Neural Network Accelerator with Adaptive Parallelism Scheme

An Efficient Hardware Design for Accelerating Sparse CNNs With NAS-Based Models

A Reconfigurable Accelerator for Sparse Convolutional Neural Networks.

A Precision-Scalable Deep Neural Network Accelerator with Activation Sparsity Exploitation

SparseNN: A Performance-Efficient Accelerator for Large-Scale Sparse Neural Networks

A Parallel Loading Based Accelerator for Convolution Neural Network

An Efficient Sparse CNNs Accelerator on FPGA

Cambricon-S: Addressing Irregularity in Sparse Neural Networks Through A Cooperative Software/Hardware Approach.