Abstract:Quantization is a key method for deploying deep neural networks on edge devices with limited memory and computation resources. Recent improvements in Post-Training Quantization (PTQ) methods were achieved by an additional local optimization process for learning the weight quantization rounding policy. However, a gap exists when employing network-wise optimization with small representative datasets. In this paper, we propose a new method for enhanced PTQ (EPTQ) that employs a network-wise quantization optimization process, which benefits from considering cross-layer dependencies during optimization. EPTQ enables network-wise optimization with a small representative dataset using a novel sample-layer attention score based on a label-free Hessian matrix upper bound. The label-free approach makes our method suitable for the PTQ scheme. We give a theoretical analysis for the said bound and use it to construct a knowledge distillation loss that guides the optimization to focus on the more sensitive layers and samples. In addition, we leverage the Hessian upper bound to improve the weight quantization parameters selection by focusing on the more sensitive elements in the weight tensors. Empirically, by employing EPTQ we achieve state-of-the-art results on various models, tasks, and datasets, including ImageNet classification, COCO object detection, and Pascal-VOC for semantic segmentation.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to solve the problem of limited memory and computing resources when deploying deep neural networks (DNNs) on edge devices. Specifically, the paper proposes an enhanced Post - Training Quantization (PTQ) method - EPTQ (Enhanced Post - Training Quantization) to improve the performance of the quantized model. #### Background problems 1. **Importance of quantization**: Quantization is one of the key techniques for deploying DNN models on edge devices. By converting weight and activation tensors into low - bit representations, the size and computational requirements of the model can be significantly reduced. 2. **Limitations of existing PTQ methods**: - Although PTQ methods are simple and efficient, they usually lead to a decrease in accuracy. - Existing network - level optimization methods have generalization error problems when using small - scale representative datasets. - Some methods rely on labeled data, which limits their flexibility in practical applications. #### Core problems of the paper The paper attempts to overcome the above limitations by introducing a new optimization method, in particular: - How to consider cross - layer dependencies in network - level optimization to improve the accuracy of the quantized model. - How to perform effective Hessian matrix analysis without relying on labeled data, thereby guiding the selection and optimization of quantization parameters. #### Main contributions 1. **Proposing the EPTQ method**: An optimization method based on knowledge distillation loss and sample - layer attention score (Sample - Layer Attention, SLA), which can compensate for the accuracy degradation caused by weight perturbation. 2. **Introducing the Hessian - based MSE (HMSE) metric**: Improve the selection of weight quantization parameters by minimizing the quantization error of weight tensor elements that have a greater impact on the task loss. 3. **Achieving SOTA results**: Demonstrates state - of - the - art performance on multiple tasks (image classification, object detection, semantic segmentation) and datasets (ImageNet, COCO, Pascal VOC). Through these improvements, EPTQ can significantly improve the accuracy and robustness of the quantized model while maintaining efficiency, especially when using small - scale representative datasets.

EPTQ: Enhanced Post-Training Quantization via Hessian-guided Network-wise Optimization