Abstract:Recent advancements in diffusion models, particularly the trend of architectural transformation from UNet-based Diffusion to Diffusion Transformer (DiT), have significantly improved the quality and scalability of image synthesis. Despite the incredible generative quality, the large computational requirements of these large-scale models significantly hinder the deployments in real-world scenarios. Post-training Quantization (PTQ) offers a promising solution by compressing model sizes and speeding up inference for the pretrained models while eliminating model retraining. However, we have observed the existing PTQ frameworks exclusively designed for both ViT and conventional Diffusion models fall into biased quantization and result in remarkable performance degradation. In this paper, we find that the DiTs typically exhibit considerable variance in terms of both weight and activation, which easily runs out of the limited numerical representations. To address this issue, we devise Q-DiT, which seamlessly integrates three techniques: fine-grained quantization to manage substantial variance across input channels of weights and activations, an automatic search strategy to optimize the quantization granularity and mitigate redundancies, and dynamic activation quantization to capture the activation changes across timesteps. Extensive experiments on the ImageNet dataset demonstrate the effectiveness of the proposed Q-DiT. Specifically, when quantizing DiT-XL/2 to W8A8 on ImageNet 256x256, Q-DiT achieves a remarkable reduction in FID by 1.26 compared to the baseline. Under a W4A8 setting, it maintains high fidelity in image generation, showcasing only a marginal increase in FID and setting a new benchmark for efficient, high-quality quantization in diffusion transformers. Code is available at \href{<a class="link-external link-https" href="https://github.com/Juanerx/Q-DiT" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/Juanerx/Q-DiT" rel="external noopener nofollow">this https URL</a>}.

Frequency Domain Distillation for Data-Free Quantization of Vision Transformer.

Quantized Feature Distillation for Network Quantization

Multi-Dimension Compression of Feed-Forward Network in Vision Transformers

Post-Training Quantization for Vision Transformer in Transformed Domain.

Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer

MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity

Semantics Prompting Data-Free Quantization for Low-Bit Vision Transformers

Quantformer: Learning Extremely Low-precision Vision Transformers

Towards Accurate Post-Training Quantization for Vision Transformer

Q-ViT: Fully Differentiable Quantization for Vision Transformer

BFD: Binarized Frequency-enhanced Distillation for Vision Transformer

ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers

PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization

SpectralKD: Understanding and Optimizing Vision Transformer Distillation through Spectral Analysis

Bi-ViT: Pushing the Limit of Vision Transformer Quantization

Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation

DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers

Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers

ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation

Oscillation-free Quantization for Low-bit Vision Transformers

Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey