SFDDM: Single-fold Distillation for Diffusion models

Chi Hong,Jiyue Huang,Robert Birke,Dick Epema,Stefanie Roos,Lydia Y. Chen

2024-05-24

Abstract:While diffusion models effectively generate remarkable synthetic images, a key limitation is the inference inefficiency, requiring numerous sampling steps. To accelerate inference and maintain high-quality synthesis, teacher-student distillation is applied to compress the diffusion models in a progressive and binary manner by retraining, e.g., reducing the 1024-step model to a 128-step model in 3 folds. In this paper, we propose a single-fold distillation algorithm, SFDDM, which can flexibly compress the teacher diffusion model into a student model of any desired step, based on reparameterization of the intermediate inputs from the teacher model. To train the student diffusion, we minimize not only the output distance but also the distribution of the hidden variables between the teacher and student model. Extensive experiments on four datasets demonstrate that our student model trained by the proposed SFDDM is able to sample high-quality data with steps reduced to as little as approximately 1%, thus, trading off inference time. Our remarkable performance highlights that SFDDM effectively transfers knowledge in single-fold distillation, achieving semantic consistency and meaningful image interpolation.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

This paper focuses on improving the inference efficiency of diffusion models, which require a large number of sampling steps to generate high-quality images. The paper proposes a new algorithm called Single-fold Distillation (SFDDM) with the aim of compressing the teacher diffusion model into a student model with arbitrary steps efficiently, thus reducing the inference time.

In the current methods, such as step-wise distillation, the teacher model is reduced to the student model through multiple rounds of binary folding and retraining, leading to high distillation time and specific step restrictions. SFDDM, on the other hand, allows for flexible compression of the teacher model to any number of steps by reparameterizing the intermediate inputs of the teacher model, while minimizing the differences between the outputs and hidden variables to maintain image synthesis quality.

Experiments show that the student models trained using SFDDM can generate high-quality data while significantly reducing the sampling steps (approximately 1%). It also demonstrates the knowledge transfer effects of SFDDM in single-fold distillation, achieving semantic consistency and meaningful image interpolation.

In summary, this paper addresses the problem of effectively reducing the inference steps of diffusion models while maintaining their image generation quality. The proposed SFDDM algorithm achieves this goal through a single distillation process.