Abstract:Diffusion models have demonstrated remarkable capabilities in image synthesis and related generative tasks. Nevertheless, their practicality for real-world applications is constrained by substantial computational costs and latency issues. Quantization is a dominant way to compress and accelerate diffusion models, where post-training quantization (PTQ) and quantization-aware training (QAT) are two main approaches, each bearing its own properties. While PTQ exhibits efficiency in terms of both time and data usage, it may lead to diminished performance in low bit-width. On the other hand, QAT can alleviate performance degradation but comes with substantial demands on computational and data resources. In this paper, we introduce a data-free and parameter-efficient fine-tuning framework for low-bit diffusion models, dubbed EfficientDM, to achieve QAT-level performance with PTQ-like efficiency. Specifically, we propose a quantization-aware variant of the low-rank adapter (QALoRA) that can be merged with model weights and jointly quantized to low bit-width. The fine-tuning process distills the denoising capabilities of the full-precision model into its quantized counterpart, eliminating the requirement for training data. We also introduce scale-aware optimization and temporal learned step-size quantization to further enhance performance. Extensive experimental results demonstrate that our method significantly outperforms previous PTQ-based diffusion models while maintaining similar time and data efficiency. Specifically, there is only a 0.05 sFID increase when quantizing both weights and activations of LDM-4 to 4-bit on ImageNet 256x256. Compared to QAT-based methods, our EfficientDM also boasts a 16.2x faster quantization speed with comparable generation quality. Code is available at \href{<a class="link-external link-https" href="https://github.com/ThisisBillhe/EfficientDM" rel="external noopener nofollow">this https URL</a>}{this hrl}.

A Fully Quantized Training Accelerator for Diffusion Network with Tensor Type & Noise Strength Aware Precision Scheduling

Hessian-based Mixed-Precision Quantization with Transition Aware Training for Neural Networks

Q-Diffusion: Quantizing Diffusion Models

Exploiting Retraining-Based Mixed-Precision Quantization for Low-Cost DNN Accelerator Design

A fine-grained mixed precision DNN accelerator using a two-stage big-little core RISC-V MCU.

Block-Wise Dynamic-Precision Neural Network Training Acceleration via Online Quantization Sensitivity Analytics

Improved Noise Schedule for Diffusion Training

Towards Accurate Post-training Quantization for Diffusion Models

Temporal Feature Matters: A Framework for Diffusion Model Quantization

Structured Dynamic Precision for Deep Neural Networks Quantization

EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models

TMPQ-DM: Joint Timestep Reduction and Quantization Precision Selection for Efficient Diffusion Models

PTQD: Accurate Post-Training Quantization for Diffusion Models

DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation

Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models

Training Quantized Network with Auxiliary Gradient Module.

Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures

QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning

Gradient-based Automatic Mixed Precision Quantization for Neural Networks On-Chip

QNCD: Quantization Noise Correction for Diffusion Models