Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models

Tuomas Kynkäänniemi,Miika Aittala,Tero Karras,Samuli Laine,Timo Aila,Jaakko Lehtinen
2024-04-11
Abstract:Guidance is a crucial technique for extracting the best performance out of image-generating diffusion models. Traditionally, a constant guidance weight has been applied throughout the sampling chain of an image. We show that guidance is clearly harmful toward the beginning of the chain (high noise levels), largely unnecessary toward the end (low noise levels), and only beneficial in the middle. We thus restrict it to a specific range of noise levels, improving both the inference speed and result quality. This limited guidance interval improves the record FID in ImageNet-512 significantly, from 1.81 to 1.40. We show that it is quantitatively and qualitatively beneficial across different sampler parameters, network architectures, and datasets, including the large-scale setting of Stable Diffusion XL. We thus suggest exposing the guidance interval as a hyperparameter in all diffusion models that use guidance.
Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning,Neural and Evolutionary Computing
What problem does this paper attempt to address?
This paper explores the application of guidance strategies in diffusion models. Traditionally, a constant guidance weight is applied throughout the image generation process. However, research has found that this approach is harmful in the early stages of high noise levels, unnecessary in the later stages of low noise levels, and only beneficial in the intermediate noise levels. Therefore, the paper proposes limiting the guidance to specific noise level ranges to improve inference speed and result quality. This approach significantly improves the recorded FID score from 1.81 to 1.40 on the ImageNet-512 dataset. The study also shows that the optimized guidance interval is beneficial for different sampler parameters, network architectures, and datasets, including large-scale Stable Diffusion XL. The paper points out through analysis that guidance reduces diversity and leads to templated images at high noise levels, helps determine image features at intermediate noise levels, and has little effect at low noise levels. Therefore, applying guidance only at the intermediate noise level can avoid adverse effects while reducing computational costs. The authors recommend publicly specifying the guidance interval as a hyperparameter in all diffusion models that use guidance. Experimental results show that the optimized guidance interval not only improves the quality of image generation in quantifiable metrics but also enhances visual effects, making it applicable to different settings. The paper provides code for further research and application.