Hardware-Centric AutoML for Mixed-Precision Quantization

Kuan Wang,Zhijian Liu,Yujun Lin,Ji Lin,Song Han

DOI: https://doi.org/10.1007/s11263-020-01339-6

2020-08-12

Abstract:Model quantization is a widely used technique to compress and accelerate deep neural network (DNN) inference. Emergent DNN hardware accelerators begin to support mixed precision (1-8 bits) to further improve the computation efficiency, which raises a great challenge to find the optimal bitwidth for each layer: it requires domain experts to explore the vast design space trading off among accuracy, latency, energy, and model size, which is both time-consuming and sub-optimal. Conventional quantization algorithm ignores the different hardware architectures and quantizes all the layers in a uniform way. In this paper, we introduce the Hardware-Aware Automated Quantization (HAQ) framework which leverages the reinforcement learning to automatically determine the quantization policy, and we take the hardware accelerator's feedback in the design loop. Rather than relying on proxy signals such as FLOPs and model size, we employ a hardware simulator to generate direct feedback signals (latency and energy) to the RL agent. Compared with conventional methods, our framework is fully automated and can specialize the quantization policy for different neural network architectures and hardware architectures. Our framework effectively reduced the latency by 1.4-1.95x and the energy consumption by 1.9x with negligible loss of accuracy compared with the fixed bitwidth (8 bits) quantization. Our framework reveals that the optimal policies on different hardware architectures (i.e., edge and cloud architectures) under different resource constraints (i.e., latency, energy, and model size) are drastically different. We interpreted the implication of different quantization policies, which offer insights for both neural network architecture design and hardware architecture design.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to determine the optimal quantization strategy for each layer of deep neural networks (DNN) on different hardware accelerators. Specifically, the paper focuses on how mixed - precision quantization (1 - 8 bits) can effectively balance the relationship among accuracy, latency, energy consumption and model size while improving computational efficiency. Traditional methods usually rely on the experience of domain experts to explore the huge design space, which is not only time - consuming but also usually not the optimal solution. In addition, existing quantization algorithms often ignore the differences of different hardware architectures and adopt a unified quantization method for all layers.

The paper proposes a reinforcement - learning - based framework - Hardware - Aware Automated Quantization (HAQ), which can automatically determine the quantization strategy and takes into account the feedback of specific hardware accelerators. In this way, the HAQ framework not only realizes automation, but also can provide specialized quantization strategies for different neural networks and hardware architectures, thereby effectively reducing latency and energy consumption while maintaining high accuracy. Compared with fixed - bit - width (such as 8 - bit) quantization, the HAQ framework can significantly reduce latency by 1.4 to 1.95 times and reduce energy consumption by 1.9 times under different hardware architectures, with negligible loss of accuracy.