Abstract:Recent studies on inverse problems have proposed posterior samplers that leverage the pre-trained diffusion models as powerful priors. These attempts have paved the way for using diffusion models in a wide range of inverse problems. However, the existing methods entail computationally demanding iterative sampling procedures and optimize a separate solution for each measurement, which leads to limited scalability and lack of generalization capability across unseen samples. To address these limitations, we propose a novel approach, Diffusion prior-based Amortized Variational Inference (DAVI) that solves inverse problems with a diffusion prior from an amortized variational inference perspective. Specifically, instead of separate measurement-wise optimization, our amortized inference learns a function that directly maps measurements to the implicit posterior distributions of corresponding clean data, enabling a single-step posterior sampling even for unseen measurements. Extensive experiments on image restoration tasks, e.g., Gaussian deblur, 4$\times$ super-resolution, and box inpainting with two benchmark datasets, demonstrate our approach's superior performance over strong baselines. Code is available at <a class="link-external link-https" href="https://github.com/mlvlab/DAVI" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address the Noisy Inverse Problems, particularly in the context of image restoration tasks. Specifically, the noisy inverse problem involves estimating the original clean data $ x_0 $ from a noisy observation $ y $. This problem is crucial in many practical applications, such as image restoration, medical imaging, and astronomy. Existing methods typically require computationally intensive iterative sampling processes and optimize each measurement individually, leading to limited scalability and insufficient generalization capabilities. To overcome these limitations, the authors propose a novel method—Diffusion Prior-based Amortized Variational Inference (DAVI). ### Main Contributions 1. **Proposed New Method**: DAVI approaches the problem from the perspective of amortized variational inference, utilizing a pre-trained diffusion model as the prior distribution to solve the inverse problem. 2. **Efficient Posterior Sampling**: Achieves efficient posterior sampling through a single neural network evaluation and can generalize to unseen measurements without any optimization steps. 3. **Introduced Perturbed Posterior Bridge (PPB)**: Provides intermediate measurements to further enhance the method's generalization capability. 4. **Experimental Validation**: Conducted extensive experiments on image restoration tasks (such as Gaussian deblurring, 4x super-resolution, and inpainting), demonstrating the superior performance of the method compared to strong baseline methods. ### Method Overview - **Amortized Variational Inference Framework**: DAVI learns a function that maps measurements directly to the implicit posterior distribution of the corresponding clean data, enabling efficient one-step posterior sampling. - **Optimization Objective**: Optimizes this function by minimizing the Kullback-Leibler (KL) divergence between the implicit distribution and the true posterior distribution. - **Perturbed Posterior Bridge**: Enhances the model's generalization capability by introducing intermediate measurements. ### Experimental Results - **Quantitative Analysis**: Experimental results on the FFHQ and ImageNet datasets show that DAVI outperforms existing methods in all tasks, particularly excelling in FID and LPIPS metrics. - **Qualitative Analysis**: Visual comparisons indicate that DAVI can generate more realistic and detailed images while maintaining measurement consistency. - **Robustness**: DAVI demonstrates good robustness to unknown noise scales $ \sigma_y $, whereas other methods show significant performance degradation under unknown noise scales. ### Conclusion The DAVI method proposed in this paper makes significant progress in solving noisy inverse problems, particularly excelling in image restoration tasks. Through efficient one-step posterior sampling and strong generalization capabilities, DAVI provides a new solution for practical applications.

Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems

A Variational Perspective on Solving Inverse Problems with Diffusion Models

Amortized Posterior Sampling with Diffusion Prior Distillation

Variational Diffusion Posterior Sampling with Midpoint Guidance

Diffusion Priors for Variational Likelihood Estimation and Image Denoising

Diffusion Posterior Sampling for General Noisy Inverse Problems

Diffusion State-Guided Projected Gradient for Inverse Problems

Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems

Unleashing the Denoising Capability of Diffusion Prior for Solving Inverse Problems

Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors

Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing

Tweedie Moment Projected Diffusions For Inverse Problems

Improving Diffusion Models for Inverse Problems Using Optimal Posterior Covariance

Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements

Solving Video Inverse Problems Using Image Diffusion Models

Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering

Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling

VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision