Abstract:Inverse problems arise in a multitude of applications, where the goal is to recover a clean signal from noisy and possibly (non)linear observations. The difficulty of a reconstruction problem depends on multiple factors, such as the ground truth signal structure, the severity of the degradation and the complex interactions between the above. This results in natural sample-by-sample variation in the difficulty of a reconstruction problem. Our key observation is that most existing inverse problem solvers lack the ability to adapt their compute power to the difficulty of the reconstruction task, resulting in subpar performance and wasteful resource allocation. We propose a novel method, $\textit{severity encoding}$, to estimate the degradation severity of corrupted signals in the latent space of an autoencoder. We show that the estimated severity has strong correlation with the true corruption level and can provide useful hints on the difficulty of reconstruction problems on a sample-by-sample basis. Furthermore, we propose a reconstruction method based on latent diffusion models that leverages the predicted degradation severities to fine-tune the reverse diffusion sampling trajectory and thus achieve sample-adaptive inference times. Our framework, Flash-Diffusion, acts as a wrapper that can be combined with any latent diffusion-based baseline solver, imbuing it with sample-adaptivity and acceleration. We perform experiments on both linear and nonlinear inverse problems and demonstrate that our technique greatly improves the performance of the baseline solver and achieves up to $10\times$ acceleration in mean sampling speed.

Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models

Accelerating Diffusion Models for Inverse Problems through Shortcut Sampling

Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models trained on Corrupted Data

DiracDiffusion: Denoising and Incremental Reconstruction with Assured Data-Consistency

Enhancing Diffusion Posterior Sampling for Inverse Problems by Integrating Crafted Measurements

Solving Video Inverse Problems Using Image Diffusion Models

ADIR: Adaptive Diffusion for Image Reconstruction

Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models

Adaptive Sampling and Reconstruction for Gradient-Domain Rendering

Diffusion adaptation framework for compressive sensing reconstruction

GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration

A Variational Perspective on Solving Inverse Problems with Diffusion Models

Diffusion Posterior Sampling for General Noisy Inverse Problems

Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing

Tweedie Moment Projected Diffusions For Inverse Problems

Adaptive Diffusion Priors for Accelerated MRI Reconstruction

Diffusion Model Based Posterior Sampling for Noisy Linear Inverse Problems

Fast Samplers for Inverse Problems in Iterative Refinement Models

Adaptive Compressed Sensing with Diffusion-Based Posterior Sampling

Solving Inverse Problems via Diffusion Optimal Control

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision