Abstract:Research on loss surface geometry, such as Sharpness-Aware Minimization (SAM), shows that flatter minima improve generalization. Recent studies further reveal that flatter minima can also reduce the domain generalization (DG) gap. However, existing flatness-based DG techniques predominantly operate within a full-precision training process, which is impractical for deployment on resource-constrained edge devices that typically rely on lower bit-width representations (e.g., 4 bits, 3 bits). Consequently, low-precision quantization-aware training is critical for optimizing these techniques in real-world applications. In this paper, we observe a significant degradation in performance when applying state-of-the-art DG-SAM methods to quantized models, suggesting that current approaches fail to preserve generalizability during the low-precision training process. To address this limitation, we propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG. Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization, where the task loss and smoothness loss induce conflicting gradients for the scaling factors of quantizers, with certain layers exhibiting opposing gradient directions. This conflict renders the optimization of quantized weights highly unstable. To mitigate this, we further introduce a mechanism to quantify gradient inconsistencies and selectively freeze the gradients of scaling factors, thereby stabilizing the training process and enhancing out-of-domain generalization. Extensive experiments validate the effectiveness of the proposed GAQAT framework. On PACS, our 3-bit and 4-bit models outperform direct DG-QAT integration by up to 4.5%. On DomainNet, the 4-bit model achieves near-lossless performance compared to full precision, with improvements of 1.39% (4-bit) and 1.06% (3-bit) over the SOTA QAT baseline.

QGen: On the Ability to Generalize in Quantization Aware Training

Hessian-based Mixed-Precision Quantization with Transition Aware Training for Neural Networks

Optimizing Quantized Neural Networks in a Weak Curvature Manifold

QT-DoG: Quantization-aware Training for Domain Generalization

Deep quantization generative networks

Constraint Guided Model Quantization of Neural Networks

Strong Generalization in Quantum Neural Networks

Long-Range Zero-Shot Generative Deep Network Quantization

Quantization Networks

GAQAT: gradient-adaptive quantization-aware training for domain generalization

GenQ: Quantization in Low Data Regimes with Generative Synthetic Data

QGAN: Quantized Generative Adversarial Networks.

Generative Low-Bitwidth Data Free Quantization

Intriguing Properties of Quantization at Scale

Quantization of Generative Adversarial Networks for Efficient Inference: a Methodological Study

Why Quantization Improves Generalization: NTK of Binary Weight Neural Networks

Gradient-based Automatic Mixed Precision Quantization for Neural Networks On-Chip

Automatic low-bit hybrid quantization of neural networks through meta learning

Post-training Quantization or Quantization-aware Training? That is the Question

Diverse Sample Generation: Pushing the Limit of Generative Data-Free Quantization

A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics