Abstract:Transfer learning, which transfers knowledge from source datasets to target datasets, is practical for adaptive deep neural network (DNN) applications. When considering user privacy and communication bandwidth issues, edge devices’ training is essential for transfer learning. Nevertheless, training requires repeating feedforward (FF), backpropagation (BP), and weight gradient (WG) millions of times, introducing prohibitive computation for edge devices. A promising method to reduce training computation is sparse DNN training (SDT), which dynamically prunes weights during training iterations and performs FF, BP, and WG only with unpruned weights. However, SDT suffers implicit redundancy and reuse imbalance for convolution layers. Besides, it turns bottlenecks into batch normalization (BN) layers. Therefore, it is challenging to achieve energy-efficient SDT computing. This article proposes a processor, Trainer, solving the above challenges with three features. First, a speculation mechanism removes implicit redundant operations, which have nonzeros’ input, weight, or output, but are ineffective for training. Second, a dynamic sparsity adaptive dataflow tackles the reuse imbalance, improving energy efficiency (EE) for dynamic sparse convolution in SDT. Third, a computational dependence decoupled BN unit eliminates BN’s repeated data access to reduce training energy and time. Trainer is fabricated in 28-nm CMOS technology and occupies 20.96 mm2 of area. It achieves a peak EE of 173.28TFLOPS/W@FP16 (276.55TFLOPS/W@FP8) for a 90% activation sparsity and 90% weight sparsity. The sparsity to EE conversion ratio is 80.9, outperforming the previous work by 1.55 $\times $ . When training a ResNet18 model with SDT, Trainer reduces energy by 2.23 $\times $ and time by 1.76 $\times $ than the state-of-the-art sparse training processor.

LPE: Logarithm Posit Processing Element for Energy-Efficient Edge-Device Training

PL-NPU: an Energy-Efficient Edge-Device DNN Training Processor with Posit-Based Logarithm-Domain Computing

A Logarithmic Floating-Point Multiplier for the Efficient Training of Neural Networks

CPE: an Energy-Efficient Edge-Device Training with Multi-dimensional Compression Mechanism

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Deep Learning Training on the Edge with Low-Precision Posits

A 28nm 1.07TFLOPS/mm<sup>2</sup> Dynamic-Precision Training Processor with Online Dynamic Execution and Multi- Level-Aligned Block-FP Processing

TPE: A High-Performance Edge-Device Inference with Multi-level Transformational Mechanism

Trainer: an Energy-Efficient Edge-Device Training Processor Supporting Dynamic Weight Pruning

A 28nm 276.55TFLOPS/W Sparse Deep-Neural-Network Training Processor with Implicit Redundancy Speculation and Batch Normalization Reformulation

Training Deep Neural Networks with 8-bit Floating Point Numbers

3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration

Resource Allocation for Stable LLM Training in Mobile Edge Computing

Hardware-Efficient Logarithmic Floating-Point Multipliers for Error-Tolerant Applications

EF-Train: Enable Efficient On-device CNN Training on FPGA Through Data Reshaping for Online Adaptation or Personalization

Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks

Accelerating DNN Training in Wireless Federated Edge Learning Systems

A 28nm 128TFLOPS/W Computing-In-Memory Engine Supporting One-Shot Floating-Point NN Inference and On-Device Fine-Tuning for Edge AI

A Design Framework for Hardware-Efficient Logarithmic Floating-Point Multipliers

RCPE: an Excellent Performance Training Processor with RISC-V Based Compression Mechanism

Random resistive memory-based deep extreme point learning machine for unified visual processing