Abstract:Sampling from the posterior distribution poses a major computational challenge in solving inverse problems using latent diffusion models. Common methods rely on Tweedie's first-order moments, which are known to induce a quality-limiting bias. Existing second-order approximations are impractical due to prohibitive computational costs, making standard reverse diffusion processes intractable for posterior sampling. This paper introduces Second-order Tweedie sampler from Surrogate Loss (STSL), a novel sampler that offers efficiency comparable to first-order Tweedie with a tractable reverse process using second-order approximation. Our theoretical results reveal that the second-order approximation is lower bounded by our surrogate loss that only requires $O(1)$ compute using the trace of the Hessian, and by the lower bound we derive a new drift term to make the reverse process tractable. Our method surpasses SoTA solvers PSLD and P2L, achieving 4X and 8X reduction in neural function evaluations, respectively, while notably enhancing sampling quality on FFHQ, ImageNet, and COCO benchmarks. In addition, we show STSL extends to text-guided image editing and addresses residual distortions present from corrupted images in leading text-guided image editing methods. To our best knowledge, this is the first work to offer an efficient second-order approximation in solving inverse problems using latent diffusion and editing real-world images with corruptions.

What problem does this paper attempt to address?

This paper attempts to address the main computational challenges encountered when using Latent Diffusion Models (LDMs) to solve inverse problems, especially the problem of sampling from the posterior distribution. Specifically: 1. **Limitations of Existing Methods**: - Common methods rely on Tweedie's first - order moment, which leads to quality - limited bias. - Existing second - order approximation methods are infeasible due to high computational costs, making the standard reverse diffusion process infeasible in posterior sampling. 2. **New Method Proposed in the Paper**: - Introduced the Second - order Tweedie sampler from Surrogate Loss (STSL), a new sampler that can achieve a feasible reverse process using second - order approximation while maintaining an efficiency comparable to that of the first - order Tweedie. - Through theoretical analysis, the paper reveals that the second - order approximation can be lower - bounded by a surrogate loss function requiring only O(1) computational complexity, thereby deriving a new drift term and making the reverse process feasible. - The performance of STSL on multiple benchmark datasets (such as FFHQ, ImageNet, and COCO) exceeds that of existing state - of - the - art methods (such as PSLD and P2L), reducing the number of neural function evaluations by 4 and 8 times respectively, and significantly improving the sampling quality. 3. **Scope of Application**: - The paper not only demonstrates the superior performance of STSL in inverse problem tasks such as image deblurring, super - resolution, Gaussian deblurring, and image inpainting, but also extends to text - guided image editing, especially when dealing with noisy images. 4. **Main Contributions**: - Proposed an efficient second - order approximation method that uses the Tweedie formula to reduce the bias in first - order samplers. - Introduced a new framework for high - fidelity image editing in real - world noisy environments. - Verified through extensive experiments the excellent performance in solving inverse problems and achieving high - fidelity text - guided image editing. In summary, this paper aims to solve the computational efficiency and quality limitations of existing methods in solving inverse problems by introducing the STSL method, especially in handling complex image tasks.

Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusion

Tweedie Moment Projected Diffusions For Inverse Problems

Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models

Think Twice Before You Act: Improving Inverse Problem Solving With MCMC

Particle-Filtering-based Latent Diffusion for Inverse Problems

Reverse Transition Kernel: A Flexible Framework to Accelerate Diffusion Inference

Prompt-tuning latent diffusion models for inverse problems

Solving Inverse Problems via Diffusion Optimal Control

Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling

Solving General Noisy Inverse Problem via Posterior Sampling: A Policy Gradient Viewpoint

Diffusion Posterior Sampling for General Noisy Inverse Problems

Improving Decoupled Posterior Sampling for Inverse Problems using Data Consistency Constraint

A Variational Perspective on Solving Inverse Problems with Diffusion Models

Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems

Sequential Posterior Sampling with Diffusion Models

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements

Solving Linear Inverse Problems Using Higher-Order Annealed Langevin Diffusion

Blind Inversion using Latent Diffusion Priors

Fast Samplers for Inverse Problems in Iterative Refinement Models