Abstract:Diffusion models have demonstrated remarkable capabilities in image synthesis and related generative tasks. Nevertheless, their practicality for real-world applications is constrained by substantial computational costs and latency issues. Quantization is a dominant way to compress and accelerate diffusion models, where post-training quantization (PTQ) and quantization-aware training (QAT) are two main approaches, each bearing its own properties. While PTQ exhibits efficiency in terms of both time and data usage, it may lead to diminished performance in low bit-width. On the other hand, QAT can alleviate performance degradation but comes with substantial demands on computational and data resources. In this paper, we introduce a data-free and parameter-efficient fine-tuning framework for low-bit diffusion models, dubbed EfficientDM, to achieve QAT-level performance with PTQ-like efficiency. Specifically, we propose a quantization-aware variant of the low-rank adapter (QALoRA) that can be merged with model weights and jointly quantized to low bit-width. The fine-tuning process distills the denoising capabilities of the full-precision model into its quantized counterpart, eliminating the requirement for training data. We also introduce scale-aware optimization and temporal learned step-size quantization to further enhance performance. Extensive experimental results demonstrate that our method significantly outperforms previous PTQ-based diffusion models while maintaining similar time and data efficiency. Specifically, there is only a 0.05 sFID increase when quantizing both weights and activations of LDM-4 to 4-bit on ImageNet 256x256. Compared to QAT-based methods, our EfficientDM also boasts a 16.2x faster quantization speed with comparable generation quality. Code is available at \href{<a class="link-external link-https" href="https://github.com/ThisisBillhe/EfficientDM" rel="external noopener nofollow">this https URL</a>}{this hrl}.

MPQ-Diff: Mixed Precision Quantization for Diffusion Models

MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models

Hessian-based Mixed-Precision Quantization with Transition Aware Training for Neural Networks

PQD: Post-training Quantization for Efficient Diffusion Models

Q-DM: an Efficient Low-bit Quantized Diffusion Model

PTQD: Accurate Post-Training Quantization for Diffusion Models

EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models

Pse: Mixed Quantization Framework of Neural Networks for Efficient Deployment

Q-Diffusion: Quantizing Diffusion Models

TMPQ-DM: Joint Timestep Reduction and Quantization Precision Selection for Efficient Diffusion Models

QNCD: Quantization Noise Correction for Diffusion Models

Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models

MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization

Mixed-Precision Quantization with Cross-Layer Dependencies

CSMPQ:Class Separability Based Mixed-Precision Quantization

CSMPQ: Class Separability Based Mixed-Precision Quantization.

QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning

TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models

Qua$^2$SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models

EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models

TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models