Abstract:When deploying deep neural networks (DNNs), edge devices training is practical to improve model adaptivity for various user-specific scenarios while avoiding privacy disclosure. However, the training computation is intolerable for edge devices. It inspires sparse DNN training (SDT) into the limelight, which reduces training computation by dynamic weight pruning. Generally, SDT has two strategies based on the pruning granularity: the structured or the unstructured. Unfortunately, both of them suffer from limited training efficiency due to the gap between pruning granularity and hardware implementation. The former is hardware-friendly but has a low pruning ratio, indicating limited computation reduction. The latter has a high pruning ratio, but the unbalanced workload decreases utilization and irregular sparsity distribution causes considerable sparsity processing overhead. This paper proposes a software-hardware co- design to bridge the gap for improving the efficiency of SDT. On the algorithm side, a sub-structured pruning method, achieved with hybrid shape-wise and line-wise pruning, generates a high sparsity ratio and keeps the hardware-friendly property. On the hardware side, a sub-structured weight processing unit (SWPU) effectively handles the hybrid sparsity with three techniques. First, SWPU dynamically reorders the computation sequence with hamming-distance-based clustering, balancing the irregular workload. Second, SWPU performs runtime scheduling by exploiting the feature of sub-structured sparse convolution through a detect-before-load controller, which skips redundant memory access and sparsity processing. Third, SWPU performs sparse convolution by compressing operands with spatial disconnect log-based routing and recovers their location with bi-directional switching, avoiding the power-consumed routing logic. Synthesized with 28nm CMOS technology, SWPU can enable 0.56V-to-1.0V supply voltage with a maximum frequency of 675 MHz. It achieves a 50.1% higher pruning ratio than structured pruning and $1.53\times $ higher energy efficiency than unstructured pruning. The peak energy efficiency of SWPU is 126.04TFLOPS/W, outperforming the state-of-the-art training processor by $1.67\times $ . When training a ResNet-18 model, SWPU reduces $3.72\times $ energy and offers $4.69\times $ speedup than previous sparse training processors.

An Efficient Hardware Architecture for DNN Training by Exploiting Triple Sparsity

High-performance Reconfigurable DNN Accelerator on a Bandwidth-limited Embedded System

THETA: A High-Efficiency Training Accelerator for DNNs with Triple-Side Sparsity Exploration

SparseNN: an Energy-Efficient Neural Network Accelerator Exploiting Input and Output Sparsity

Trainer: an Energy-Efficient Edge-Device Training Processor Supporting Dynamic Weight Pruning

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

A 28nm 276.55TFLOPS/W Sparse Deep-Neural-Network Training Processor with Implicit Redundancy Speculation and Batch Normalization Reformulation

DQ-STP: an Efficient Sparse On-Device Training Processor Based on Low-Rank Decomposition and Quantization for DNN

Efficient Hardware Optimization Strategies For Deep Neural Networks Acceleration Chip

SWPU: A 126.04 TFLOPS/W Edge-Device Sparse DNN Training Processor with Dynamic Sub-Structured Weight Pruning

Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design

A Fine-Grained Sparse Accelerator for Multi-Precision DNN.

A 3.89-Gops/mw Scalable Recurrent Neural Network Processor with Improved Efficiency on Memory and Computation

A High Energy-Efficiency Multi-core Neuromorphic Architecture for Deep SNN Training

SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training

An FPGA-based Mix-grained Sparse Training Accelerator

Procrustes: a Dataflow and Accelerator for Sparse Deep Neural Network Training

SPAT: FPGA-based Sparsity-Optimized Spiking Neural Network Training Accelerator with Temporal Parallel Dataflow

Performance of Training Sparse Deep Neural Networks on GPUs

Eyelet: A Cross-Mesh NoC-Based Fine-Grained Sparse CNN Accelerator for Spatio-Temporal Parallel Computing Optimization

A Precision-Scalable Deep Neural Network Accelerator with Activation Sparsity Exploitation