Abstract:Transfer learning, which transfers knowledge from source datasets to target datasets, is practical for adaptive deep neural network (DNN) applications. When considering user privacy and communication bandwidth issues, edge devices’ training is essential for transfer learning. Nevertheless, training requires repeating feedforward (FF), backpropagation (BP), and weight gradient (WG) millions of times, introducing prohibitive computation for edge devices. A promising method to reduce training computation is sparse DNN training (SDT), which dynamically prunes weights during training iterations and performs FF, BP, and WG only with unpruned weights. However, SDT suffers implicit redundancy and reuse imbalance for convolution layers. Besides, it turns bottlenecks into batch normalization (BN) layers. Therefore, it is challenging to achieve energy-efficient SDT computing. This article proposes a processor, Trainer, solving the above challenges with three features. First, a speculation mechanism removes implicit redundant operations, which have nonzeros’ input, weight, or output, but are ineffective for training. Second, a dynamic sparsity adaptive dataflow tackles the reuse imbalance, improving energy efficiency (EE) for dynamic sparse convolution in SDT. Third, a computational dependence decoupled BN unit eliminates BN’s repeated data access to reduce training energy and time. Trainer is fabricated in 28-nm CMOS technology and occupies 20.96 mm2 of area. It achieves a peak EE of 173.28TFLOPS/W@FP16 (276.55TFLOPS/W@FP8) for a 90% activation sparsity and 90% weight sparsity. The sparsity to EE conversion ratio is 80.9, outperforming the previous work by 1.55 $\times $ . When training a ResNet18 model with SDT, Trainer reduces energy by 2.23 $\times $ and time by 1.76 $\times $ than the state-of-the-art sparse training processor.

CPE: an Energy-Efficient Edge-Device Training with Multi-dimensional Compression Mechanism

RCPE: an Excellent Performance Training Processor with RISC-V Based Compression Mechanism

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

TPE: A High-Performance Edge-Device Inference with Multi-level Transformational Mechanism

PL-NPU: an Energy-Efficient Edge-Device DNN Training Processor with Posit-Based Logarithm-Domain Computing

A 3.89-Gops/mw Scalable Recurrent Neural Network Processor with Improved Efficiency on Memory and Computation

An Efficient Hardware Architecture for DNN Training by Exploiting Triple Sparsity

Trainer: an Energy-Efficient Edge-Device Training Processor Supporting Dynamic Weight Pruning

A 28nm 276.55TFLOPS/W Sparse Deep-Neural-Network Training Processor with Implicit Redundancy Speculation and Batch Normalization Reformulation

An Emerging NVM CIM Accelerator with Shared-Path Transpose Read and Bit-Interleaving Weight Storage for Efficient On-Chip Training in Edge Devices

DQ-STP: an Efficient Sparse On-Device Training Processor Based on Low-Rank Decomposition and Quantization for DNN

LPE: Logarithm Posit Processing Element for Energy-Efficient Edge-Device Training

3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration

CAP: Communication-aware Automated Parallelization for Deep Learning Inference on CMP Architectures

THETA: A High-Efficiency Training Accelerator for DNNs with Triple-Side Sparsity Exploration

COMB-MCM: Computing-on-Memory-Boundary NN Processor with Bipolar Bitwise Sparsity Optimization for Scalable Multi-Chiplet-Module Edge Machine Learning.

A Precision-Scalable Energy-Efficient Convolutional Neural Network Accelerator.

A 3D Tiled Low Power Accelerator for Convolutional Neural Network

An Energy Efficient Computing-in-Memory Accelerator With 1T2R Cell and Fully Analog Processing for Edge AI Applications

Efficient Feature Compression for Edge-Cloud Systems

EPMC: Efficient Parallel Memory Compression in Deep Neural Network Training