Abstract:Diffusion models have been proven highly effective at generating high-quality images. However, as these models grow larger, they require significantly more memory and suffer from higher latency, posing substantial challenges for deployment. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where conventional post-training quantization methods for large language models like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit quantization paradigm. Different from smoothing which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights, then employ a high-precision low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD). This process eases the quantization on both sides. However, na\"ıvely running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without the need for re-quantization. Extensive experiments on SDXL, PixArt-$\Sigma$, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5$\times$, achieving 3.0$\times$ speedup over the 4-bit weight-only quantized baseline on the 16GB laptop 4090 GPU, paving the way for more interactive applications on PCs. Our quantization library and inference engine are open-sourced.

SVQ: Sparse Vector Quantization for Spatiotemporal Forecasting

Noise model based ν-support vector regression with its application to short-term wind speed forecasting

Time-Aware and Sparsity-Tolerant QoS Prediction Based on Collaborative Filtering

SVSeq2Seq: An Efficient Computational Method for State Vectors in Sequence-to-Sequence Architecture Forecasting

Sparse-VQ Transformer: An FFN-Free Framework with Vector Quantization for Enhanced Time Series Forecasting

A Spatial-Temporal QoS Prediction Approach for Time-aware Web Service Recommendation.

Short-Term Load Forecasting Based On Support Vector Machine

Fuzzy support vector regressions for short-term load forecasting

A novel video thumbnail extraction method using spatiotemporal vector quantization.

S-HR-VQVAE: Sequential Hierarchical Residual Learning Vector Quantized Variational Autoencoder for Video Prediction

Spatiotemporal Prediction Based on Feature Classification for Multivariate Floating-Point Time Series Lossy Compression

Quantile Surfaces -- Generalizing Quantile Regression to Multivariate Targets

Twin support vector quantile regression

Online Video Quality Enhancement with Spatial-Temporal Look-up Tables

LibVQ: A Toolkit for Optimizing Vector Quantization and Efficient Neural Retrieval.

Quantum Kernel-Based Long Short-term Memory for Climate Time-Series Forecasting

SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

An adaptive spatio-temporal perception aware quantization algorithm for AVS2

Variational quantization for state space models

Spatio-Temporal Variable Selection Based Support Vector Regression for Urban Traffic Flow Prediction

TimeSQL: Improving Multivariate Time Series Forecasting with Multi-Scale Patching and Smooth Quadratic Loss