Abstract:Diffusion models that can generate high-quality data from randomly sampled Gaussian noises have become the mainstream generative method in both academia and industry. Are randomly sampled Gaussian noises equally good for diffusion models? While a large body of works tried to understand and improve diffusion models, previous works overlooked the possibility to select or optimize the sampled noise the possibility of selecting or optimizing sampled noises for improving diffusion models. In this paper, we mainly made three contributions. First, we report that not all noises are created equally for diffusion models. We are the first to hypothesize and empirically observe that the generation quality of diffusion models significantly depend on the noise inversion stability. This naturally provides us a noise selection method according to the inversion stability. Second, we further propose a novel noise optimization method that actively enhances the inversion stability of arbitrary given noises. Our method is the first one that works on noise space to generally improve generated results without fine-tuning diffusion models. Third, our extensive experiments demonstrate that the proposed noise selection and noise optimization methods both significantly improve representative diffusion models, such as SDXL and SDXL-turbo, in terms of human preference and other objective evaluation metrics. For example, the human preference winning rates of noise selection and noise optimization over the baselines can be up to 57% and 72.5%, respectively, on DrawBench.

Removing Structured Noise with Diffusion Models

Diffusion Posterior Sampling for General Noisy Inverse Problems

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

A Variational Perspective on Solving Inverse Problems with Diffusion Models

Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance

Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing

Learning Diffusion Model from Noisy Measurement using Principled Expectation-Maximization Method

Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements

Listening to the Noise: Blind Denoising with Gibbs Diffusion

Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems

Tweedie Moment Projected Diffusions For Inverse Problems

A Survey on Diffusion Models for Inverse Problems

Edge-preserving noise for diffusion models

Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

Diffusion Models With Learned Adaptive Noise

Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models

Not All Noises Are Created Equally:Diffusion Noise Selection and Optimization

GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration

A Flexible Diffusion Model