Abstract:Diffusion models have emerged as a key pillar of foundation models in visual domains. One of their critical applications is to universally solve different downstream inverse tasks via a single diffusion prior without re-training for each task. Most inverse tasks can be formulated as inferring a posterior distribution over data (e.g., a full image) given a measurement (e.g., a masked image). This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution. We show that our approach naturally leads to regularization by denoising diffusion process (RED-Diff) where denoisers at different timesteps concurrently impose different structural constraints over the image. To gauge the contribution of denoisers from different timesteps, we propose a weighting mechanism based on signal-to-noise-ratio (SNR). Our approach provides a new variational perspective for solving inverse problems with diffusion models, allowing us to formulate sampling as stochastic optimization, where one can simply apply off-the-shelf solvers with lightweight iterates. Our experiments for image restoration tasks such as inpainting and superresolution demonstrate the strengths of our method compared with state-of-the-art sampling-based diffusion models.

What problem does this paper attempt to address?

The paper primarily focuses on addressing inverse problems in the visual domain using diffusion models. Specifically, the research aims to develop a general and adaptable method that can effectively solve various downstream inverse problems without retraining the model for each specific task, particularly in the applications of image restoration and rendering. The proposed solution is based on variational inference, approximating the posterior distribution given observations by using the diffusion model as the prior distribution of the data and combining it with the measurement model as the likelihood function. This approach allows researchers to formulate the sampling process as a stochastic optimization problem, thereby leveraging existing optimizers for efficient and easily adjustable sampling. Additionally, the method introduces a weighting mechanism based on the signal-to-noise ratio (SNR) to reasonably evaluate the contributions of denoisers at different time steps. This helps to better balance the structural constraints of the image from high-level semantics to detailed levels. This method is referred to as Regularized Denoising (RED-diff). The main contributions of the paper include: 1. Proposing the RED-diff variational method, which introduces a rigorous maximum likelihood framework to alleviate the posterior score approximation issues present in recent ΠGDM and DPS methods for general inverse problems. 2. Establishing a connection with the Regularized Denoising (RED) framework, allowing sampling to be viewed as a stochastic optimization problem, thus enabling the use of existing optimization algorithms for fast and adjustable sampling. 3. Introducing a denoising SNR-based weighting mechanism for regularization during the diffusion process. 4. Conducting extensive experiments on various linear and nonlinear inverse problems, demonstrating the superiority of RED-diff over existing methods such as ΠGDM and DPS in terms of image quality and GPU efficiency. In short, this paper aims to improve the inverse problem-solving solutions based on diffusion models through variational inference methods, enhancing the efficiency and quality of sampling.

A Variational Perspective on Solving Inverse Problems with Diffusion Models

Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

Solving Video Inverse Problems Using Image Diffusion Models

Variational Diffusion Posterior Sampling with Midpoint Guidance

Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

Diffusion State-Guided Projected Gradient for Inverse Problems

Tweedie Moment Projected Diffusions For Inverse Problems

Inverse Problems with Diffusion Models: A MAP Estimation Perspective

Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models

Conditional Variational Diffusion Models

Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance

Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements

G2D2: Gradient-guided Discrete Diffusion for image inverse problem solving

Solving Inverse Problems via Diffusion Optimal Control

Removing Structured Noise with Diffusion Models

Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors

A Survey on Diffusion Models for Inverse Problems

Diffusion Priors for Variational Likelihood Estimation and Image Denoising