Trainer: an Energy-Efficient Edge-Device Training Processor Supporting Dynamic Weight Pruning
Yang Wang,Yubin Qin,Dazheng Deng,Jingchuan Wei,Tianbao Chen,Xinhan Lin,Leibo Liu,Shaojun Wei,Shouyi Yin
DOI: https://doi.org/10.1109/jssc.2022.3174411
IF: 5.4
2022-01-01
IEEE Journal of Solid-State Circuits
Abstract:Transfer learning, which transfers knowledge from source datasets to target datasets, is practical for adaptive deep neural network (DNN) applications. When considering user privacy and communication bandwidth issues, edge devices’ training is essential for transfer learning. Nevertheless, training requires repeating feedforward (FF), backpropagation (BP), and weight gradient (WG) millions of times, introducing prohibitive computation for edge devices. A promising method to reduce training computation is sparse DNN training (SDT), which dynamically prunes weights during training iterations and performs FF, BP, and WG only with unpruned weights. However, SDT suffers implicit redundancy and reuse imbalance for convolution layers. Besides, it turns bottlenecks into batch normalization (BN) layers. Therefore, it is challenging to achieve energy-efficient SDT computing. This article proposes a processor, Trainer, solving the above challenges with three features. First, a speculation mechanism removes implicit redundant operations, which have nonzeros’ input, weight, or output, but are ineffective for training. Second, a dynamic sparsity adaptive dataflow tackles the reuse imbalance, improving energy efficiency (EE) for dynamic sparse convolution in SDT. Third, a computational dependence decoupled BN unit eliminates BN’s repeated data access to reduce training energy and time. Trainer is fabricated in 28-nm CMOS technology and occupies 20.96 mm2 of area. It achieves a peak EE of 173.28TFLOPS/W@FP16 (276.55TFLOPS/W@FP8) for a 90% activation sparsity and 90% weight sparsity. The sparsity to EE conversion ratio is 80.9, outperforming the previous work by 1.55 $\times $ . When training a ResNet18 model with SDT, Trainer reduces energy by 2.23 $\times $ and time by 1.76 $\times $ than the state-of-the-art sparse training processor.