Abstract:CNN model computation on edge devices is tightly restricted to the limited resource and power budgets, which motivates the low-bit quantization technology to compress CNN models into 4-bit or lower format to reduce the model size and increase hardware efficiency. Most current low-bit quantization methods use uniform quantization that maps weight and activation values onto evenly-distributed levels, which usually results in accuracy loss due to distribution mismatch. Meanwhile, some non-uniform quantization methods propose specialized representation that can better match various distribution shapes but are usually difficult to be efficiently accelerated on hardware. In order to achieve low-bit quantization with high accuracy and hardware efficiency, this paper proposes Universal Power-of-Two (UPoT), a novel low-bit quantization method that represents values as the addition of multiple power-of-two values selected from a series of subsets. By updating the subset contents, UPoT can provide adaptive quantization levels for various distributions. For each CNN model layer, UPoT automatically searches for the optimized distribution that minimizes the quantization error. Moreover, we design an efficient accelerator system with specifically optimized power-of-two multipliers and requantization units. Evaluations show that the proposed architecture can provide high-performance CNN inference with reduced circuit area and energy, and outperforms several mainstream CNN accelerators with higher ( $8\times $ – $65\times $ ) area efficiency and ( $2\times $ – $19\times $ ) energy efficiency. Further experiments of 4/3/2-bit quantization on ResNet18/50, MobileNet_V2 and EfficientNet models show that our UPoT can achieve high model accuracy which greatly outperform other state-of-the-art low-bit quantization methods by 0.3%–6%. The results indicate that our approach provides a highly-efficient accelerator for low-bit CNN model quantization with low hardware overheads and good model accuracy.

CNN Acceleration by Low-rank Approximation with Quantized Factors

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Single-shot Pruning and Quantization for Hardware-Friendly Neural Network Acceleration

Focused Quantization for Sparse CNNs

Convolution Without Multiplication: A General Speed Up Strategy for CNNs

Speeding-up and compression convolutional neural networks by low-rank decomposition without fine-tuning

Holistic CNN Compression Via Low-Rank Decomposition with Knowledge Transfer.

Sensitivity-based Acceleration and Compression Algorithm for Convolution Neural Network.

Transform Quantization for CNN (Convolutional Neural Network) Compression

Regularized Training Framework for Combining Pruning and Quantization to Compress Neural Networks

TEC-CNN: Towards Efficient Compressing Convolutional Neural Nets with Low-rank Tensor Decomposition

Convolutional neural networks compression with low rank and sparse tensor decompositions

Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms

Low-precision CNN Model Quantization based on Optimal Scaling Factor Estimation

Transform-Based Feature Map Compression for CNN Inference

KCNN: Kernel-wise Quantization to Remarkably Decrease Multiplications in Convolutional Neural Network.

Deep neural network compression by Tucker decomposition with nonlinear response

An Energy-and-Area-Efficient CNN Accelerator for Universal Powers-of-Two Quantization.

A Unified Approximation Framework for Compressing and Accelerating Deep Neural Networks

Compressing CNN Kernels for Videos Using Tucker Decompositions: Towards Lightweight CNN Applications

Joint Optimization of Dimension Reduction and Mixed-Precision Quantization for Activation Compression of Neural Networks