Abstract:Network quantization is a dominant paradigm of model compression. However, the abrupt changes in quantized weights during training often lead to severe loss fluctuations and result in a sharp loss landscape, making the gradients unstable and thus degrading the performance. Recently, Sharpness-Aware Minimization (SAM) has been proposed to smooth the loss landscape and improve the generalization performance of the models. Nevertheless, directly applying SAM to the quantized models can lead to perturbation mismatch or diminishment issues, resulting in suboptimal performance. In this paper, we propose a novel method, dubbed Sharpness-Aware Quantization (SAQ), to explore the effect of SAM in model compression, particularly quantization for the first time. Specifically, we first provide a unified view of quantization and SAM by treating them as introducing quantization noises and adversarial perturbations to the model weights, respectively. According to whether the noise and perturbation terms depend on each other, SAQ can be formulated into three cases, which are analyzed and compared comprehensively. Furthermore, by introducing an efficient training strategy, SAQ only incurs a little additional training overhead compared with the default optimizer (e.g., SGD or AdamW). Extensive experiments on both convolutional neural networks and Transformers across various datasets (i.e., ImageNet, CIFAR-10/100, Oxford Flowers-102, Oxford-IIIT Pets) show that SAQ improves the generalization performance of the quantized models, yielding the SOTA results in uniform quantization. For example, on ImageNet, SAQ outperforms AdamW by 1.2% on the Top-1 accuracy for 4-bit ViT-B/16. Our 4-bit ResNet-50 surpasses the previous SOTA method by 0.9% on the Top-1 accuracy.

Noise-resistant Sharpness-Aware Minimization in Deep Learning

Friendly Sharpness-Aware Minimization

Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

How Sharpness-Aware Minimization Minimizes Sharpness?

Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term

Towards Understanding Sharpness-Aware Minimization

Why Does Sharpness-Aware Minimization Generalize Better Than SGD?

Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning

Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer

On Memorization and Privacy Risks of Sharpness Aware Minimization

Adaptive Adversarial Cross-Entropy Loss for Sharpness-Aware Minimization

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training

Agnostic Sharpness-Aware Minimization

Bilateral Sharpness-Aware Minimization for Flatter Minima

Critical Influence of Overparameterization on Sharpness-aware Minimization

Sharpness-aware Quantization for Deep Neural Networks

Fundamental Convergence Analysis of Sharpness-Aware Minimization

GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization

A Universal Class of Sharpness-Aware Minimization Algorithms

An Adaptive Policy to Employ Sharpness-Aware Minimization