Abstract:When deploying deep neural networks (DNNs), edge devices training is practical to improve model adaptivity for various user-specific scenarios while avoiding privacy disclosure. However, the training computation is intolerable for edge devices. It inspires sparse DNN training (SDT) into the limelight, which reduces training computation by dynamic weight pruning. Generally, SDT has two strategies based on the pruning granularity: the structured or the unstructured. Unfortunately, both of them suffer from limited training efficiency due to the gap between pruning granularity and hardware implementation. The former is hardware-friendly but has a low pruning ratio, indicating limited computation reduction. The latter has a high pruning ratio, but the unbalanced workload decreases utilization and irregular sparsity distribution causes considerable sparsity processing overhead. This paper proposes a software-hardware co- design to bridge the gap for improving the efficiency of SDT. On the algorithm side, a sub-structured pruning method, achieved with hybrid shape-wise and line-wise pruning, generates a high sparsity ratio and keeps the hardware-friendly property. On the hardware side, a sub-structured weight processing unit (SWPU) effectively handles the hybrid sparsity with three techniques. First, SWPU dynamically reorders the computation sequence with hamming-distance-based clustering, balancing the irregular workload. Second, SWPU performs runtime scheduling by exploiting the feature of sub-structured sparse convolution through a detect-before-load controller, which skips redundant memory access and sparsity processing. Third, SWPU performs sparse convolution by compressing operands with spatial disconnect log-based routing and recovers their location with bi-directional switching, avoiding the power-consumed routing logic. Synthesized with 28nm CMOS technology, SWPU can enable 0.56V-to-1.0V supply voltage with a maximum frequency of 675 MHz. It achieves a 50.1% higher pruning ratio than structured pruning and $1.53\times $ higher energy efficiency than unstructured pruning. The peak energy efficiency of SWPU is 126.04TFLOPS/W, outperforming the state-of-the-art training processor by $1.67\times $ . When training a ResNet-18 model, SWPU reduces $3.72\times $ energy and offers $4.69\times $ speedup than previous sparse training processors.

THETA: A High-Efficiency Training Accelerator for DNNs with Triple-Side Sparsity Exploration

An Efficient Hardware Architecture for DNN Training by Exploiting Triple Sparsity

High-performance Reconfigurable DNN Accelerator on a Bandwidth-limited Embedded System

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

DQ-STP: an Efficient Sparse On-Device Training Processor Based on Low-Rank Decomposition and Quantization for DNN

Trainer: an Energy-Efficient Edge-Device Training Processor Supporting Dynamic Weight Pruning

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

Exploiting Dynamic Bit Sparsity in Activation for Deep Neural Network Acceleration.

A Precision-Scalable Deep Neural Network Accelerator with Activation Sparsity Exploitation

3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration

DSTC: Dual-Side Sparsity Tensor Core for DNNs Acceleration on Modern GPU Architectures

SWPU: A 126.04 TFLOPS/W Edge-Device Sparse DNN Training Processor with Dynamic Sub-Structured Weight Pruning

SPAT: FPGA-based Sparsity-Optimized Spiking Neural Network Training Accelerator with Temporal Parallel Dataflow

A 28nm 276.55TFLOPS/W Sparse Deep-Neural-Network Training Processor with Implicit Redundancy Speculation and Batch Normalization Reformulation

A Low-Power Accelerator for Deep Neural Networks with Enlarged Near-Zero Sparsity

T-DLA: An Open-source Deep Learning Accelerator for Ternarized DNN Models on Embedded FPGA

SparseNN: an Energy-Efficient Neural Network Accelerator Exploiting Input and Output Sparsity

An FPGA-based Mix-grained Sparse Training Accelerator

Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design

An Energy-Efficient Spiking Neural Network Accelerator Based on Spatio-Temporal Redundancy Reduction

A 3D Tiled Low Power Accelerator for Convolutional Neural Network