Abstract:The record-breaking performance of deep neural networks (DNNs) comes with heavy parameter budgets, which leads to external dynamic random access memory (DRAM) for storage. The prohibitive energy of DRAM accesses makes it nontrivial for DNN deployment on resource-constrained devices, calling for minimizing the movements of weights and data in order to improve the energy efficiency. Driven by this critical bottleneck, we present SmartDeal, a hardware-friendly algorithm framework to trade higher-cost memory storage/access for lower-cost computation, in order to aggressively boost the storage and energy efficiency, for both DNN inference and training. The core technique of SmartDeal is a novel DNN weight matrix decomposition framework with respective structural constraints on each matrix factor, carefully crafted to unleash the hardware-aware efficiency potential. Specifically, we decompose each weight tensor as the product of a small basis matrix and a large structurally sparse coefficient matrix whose nonzero elements are readily quantized to the power-of-2. The resulting sparse and readily quantized DNNs enjoy greatly reduced energy consumption in data movement as well as weight storage, while incurring minimal overhead to recover the original weights thanks to the required sparse bit-operations and cost-favorable computations. Beyond inference, we take another leap to embrace energy-efficient training, by introducing several customized techniques to address the unique roadblocks arising in training while preserving the SmartDeal structures. We also design a dedicated hardware accelerator to fully utilize the new weight structure to improve the real energy efficiency and latency performance. We conduct experiments on both vision and language tasks, with nine models, four datasets, and three settings (inference-only, adaptation, and fine-tuning). Our extensive results show that 1) being applied to inference, SmartDeal achieves up to 2.44× improvement in energy efficiency as evaluated using real hardware implementations and 2) being applied to training, SmartDeal can lead to 10.56× and 4.48× reduction in the storage and the training energy cost, respectively, with usually negligible accuracy loss, compared to state-of-the-art training baselines. Our source codes are available at: https://github.com/VITA-Group/SmartDeal.

WeightLock: A Mixed-Grained Weight Encryption Approach Using Local Decrypting Units for Ciphertext Computing in DNN Accelerators

A Convolutional Neural Network Accelerator Architecture with Fine-Granular Mixed Precision Configurability.

Enabling Secure In-Memory Neural Network Computing by Sparse Fast Gradient Encryption

WESCO: Weight-encoded Reliability and Security Co-design for In-memory Computing Systems

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

SNNGX: Securing Spiking Neural Networks with Genetic XOR Encryption on RRAM-based Neuromorphic Accelerator

Novel Ferroelectric Tunnel FinFET based Encryption-embedded Computing-in-Memory for Secure AI with High Area- and Energy-Efficiency

Layer-Wise Security Protection for Deep Neural Networks in Industrial Cyber Physical Systems

Weight Block Sparsity: Training, Compilation, and AI Engine Accelerators

DNN-Defender: An in-DRAM Deep Neural Network Defense Mechanism for Adversarial Weight Attack

Structured Compression by Weight Encryption for Unstructured Pruning and Quantization

HWGN2: Side-channel Protected Neural Networks through Secure and Private Function Evaluation

Low-latency DAE: A Configurable Lightweight Hybrid Data and Address Encryption Engine for IoT Real-time NVM Protection

Enhancing Security of Memristor Computing System Through Secure Weight Mapping

Structured Weight Matrices-Based Hardware Accelerators in Deep Neural Networks: FPGAs and ASICs

An area and energy efficient design of domain-wall memory-based deep convolutional neural networks using stochastic computing

DNN-Defender: A Victim-Focused In-DRAM Defense Mechanism for Taming Adversarial Weight Attack on DNNs

Deep-Dup: An Adversarial Weight Duplication Attack Framework to Crush Deep Neural Network in Multi-Tenant FPGA

Chaotic Weights: A Novel Approach to Protect Intellectual Property of Deep Neural Networks

SmartDeal: Remodeling Deep Network Weights for Efficient Inference and Training

Older and Wiser: The Marriage of Device Aging and Intellectual Property Protection of Deep Neural Networks