Abstract:Neural networks based on Long Short-Term Memory (LSTM) are widely deployed in latency-sensitive language and speech applications. To speed up LSTM inference, previous research proposes weight pruning techniques to reduce computational cost. Unfortunately, irregular computation and memory accesses in unrestricted sparse LSTM limit the realizable parallelism, especially when implemented on FPGA. To address this issue, some researchers propose block-based sparsity patterns to increase the regularity of sparse weight matrices, but these approaches suffer from deteriorated prediction accuracy. This work presents Bank-Balanced Sparsity (BBS), a novel sparsity pattern that can maintain model accuracy at a high sparsity level while still enable an efficient FPGA implementation. BBS partitions each weight matrix row into banks for parallel computing, while adopts fine-grained pruning inside each bank to maintain model accuracy. We develop a 3-step software-hardware co-optimization approach to apply BBS in real FPGA hardware. First, we propose a bank-balanced pruning method to induce the BBS pattern on weight matrices. Then we introduce a decoding-free sparse matrix format, Compressed Sparse Banks (CSB), that transparently exposes inter-bank parallelism in BBS to hardware. Finally, we design an FPGA accelerator that takes advantage of BBS to eliminate irregular computation and memory accesses. Implemented on Intel Arria-10 FPGA, the BBS accelerator can achieve 750.9 GOPs on sparse LSTM networks with a batch size of 1. Compared to state-of-the-art FPGA accelerators for LSTM with different compression techniques, the BBS accelerator achieves 2.3 ~ 3.7x improvement on energy efficiency and 7.0 ~ 34.4x reduction on latency with negligible loss of model accuracy.

A compression strategy to accelerate LSTM meta-learning on FPGA

A High Energy-Efficiency FPGA-Based LSTM Accelerator Architecture Design by Structured Pruning and Normalized Linear Quantization

C-LSTM: Enabling Efficient LSTM Using Structured Compression Techniques on FPGAs

E-LSTM: an Efficient Hardware Architecture for Long Short-Term Memory

Acceleration of LSTM with Structured Pruning Method on FPGA

A low-latency LSTM accelerator using balanced sparsity based on FPGA

An Fpga-Based Lstm Acceleration Engine For Deep Learning Frameworks

Implementation and Optimization of the Accelerator Based on FPGA Hardware for LSTM Network

Efficient and Effective Sparse LSTM on FPGA with Bank-Balanced Sparsity

An Instruction-Driven Batch-Based High-Performance Resource-Efficient LSTM Accelerator on FPGA

PermLSTM: A High Energy-Efficiency LSTM Accelerator Architecture

An LSTM Acceleration Engine for FPGAs Based on Caffe Framework

Design of Low Power LSTM Neural Network Accelerator Based on FPGA

F-LSTM: FPGA-Based Heterogeneous Computing Framework for Deploying LSTM-Based Algorithms

A Power-Efficient Accelerator Based on FPGAs for LSTM Network

Hardware-Oriented Compression of Long Short-Term Memory for Efficient Inference.

A Configurable FPGA Accelerator of Bi-LSTM Inference with Structured Sparsity

An FPGA-based Mix-grained Sparse Training Accelerator

FPGA-based Accelerator for Long Short-Term Memory Recurrent Neural Networks

FPGA Accelerator for CNN: an Exploration of the Kernel Structured Sparsity and Hybrid Arithmetic Computation

An SSD-MobileNet Acceleration Strategy for FPGAs Based on Network Compression and Subgraph Fusion