Denoising Diffusion Models for Plug-and-Play Image Restoration

Yuanzhi Zhu,Kai Zhang,Jingyun Liang,Jiezhang Cao,Bihan Wen,Radu Timofte,Luc Van Gool

2023-05-16

Abstract:Plug-and-play Image Restoration (IR) has been widely recognized as a flexible and interpretable method for solving various inverse problems by utilizing any off-the-shelf denoiser as the implicit image prior. However, most existing methods focus on discriminative Gaussian denoisers. Although diffusion models have shown impressive performance for high-quality image synthesis, their potential to serve as a generative denoiser prior to the plug-and-play IR methods remains to be further explored. While several other attempts have been made to adopt diffusion models for image restoration, they either fail to achieve satisfactory results or typically require an unacceptable number of Neural Function Evaluations (NFEs) during inference. This paper proposes DiffPIR, which integrates the traditional plug-and-play method into the diffusion sampling framework. Compared to plug-and-play IR methods that rely on discriminative Gaussian denoisers, DiffPIR is expected to inherit the generative ability of diffusion models. Experimental results on three representative IR tasks, including super-resolution, image deblurring, and inpainting, demonstrate that DiffPIR achieves state-of-the-art performance on both the FFHQ and ImageNet datasets in terms of reconstruction faithfulness and perceptual quality with no more than 100 NFEs. The source code is available at {\url{<a class="link-external link-https" href="https://github.com/yuanzhi-zhu/DiffPIR" rel="external noopener nofollow">this https URL</a>}}

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to address the challenges in image restoration (IR), especially how to use diffusion models as generative denoising priors in combination with the traditional plug - and - play (PnP) method to solve various low - level vision tasks. Specifically, the paper proposes a method named **DiffPIR**, which integrates the traditional plug - and - play method in the diffusion sampling framework to solve problems such as image super - resolution, deblurring and inpainting.

### Main problem background

1. **Plug - and - Play Image Restoration Method (PnP - IR)**
- The PnP - IR method separates the data term and the prior term and uses any off - the - shelf denoiser as an implicit image prior to flexibly and interpretably solve various inverse problems.
- Most of the existing methods mainly focus on discriminative Gaussian denoisers, while diffusion models perform excellently in high - quality image generation, but their potential as generative denoising priors has not been fully explored.

2. **Diffusion Models**
- Diffusion models map data to noise through the forward diffusion process and then gradually remove Gaussian noise in the reverse process to generate high - quality images.
- Although diffusion models perform well in image generation tasks, when applied to image restoration, they either cannot achieve satisfactory results or require a large number of neural function evaluations (NFEs).

### Main contributions of the paper

1. **Proposing the DiffPIR method**
- DiffPIR combines the traditional PnP - IR method with the diffusion sampling framework and inherits the generative ability of diffusion models.
- By adding measurement information in each reverse diffusion step, DiffPIR can efficiently perform image restoration while maintaining high - quality generation.

2. **Experimental verification**
- The experimental results show that DiffPIR not only performs excellently in reconstruction fidelity but also reaches the state - of - the - art level in perceptual quality in super - resolution, deblurring and inpainting tasks on the FFHQ and ImageNet datasets.
- Compared with existing methods, DiffPIR can achieve high - quality image restoration with no more than 100 NFEs.

### Specific problems and solutions

1. **Optimization of inverse problems**
- Through variable splitting algorithms (such as the alternating direction multiplier method ADMM and the half - quadratic splitting HQS), the PnP - IR method integrates the Gaussian denoiser into the iterative process, improving performance and convergence speed.
- Based on this, DiffPIR uses the diffusion model as a generative denoising prior to solve the limitations of traditional denoisers in dealing with ill - posed inverse problems.

2. **Generative ability of diffusion models**
- Diffusion models can model complex distributions, so they have better performance in dealing with ill - posed inverse problems.
- DiffPIR gradually adds measurement information in the reverse diffusion process to ensure that the generated image conforms to the prior distribution and also satisfies the given measurement constraints.

3. **Efficient sampling strategy**
- In order to improve sampling efficiency, DiffPIR introduces a hyperparameter ζ to balance the predicted noise and the random noise, making the sampling process more deterministic.
- The experimental results show that DiffPIR can achieve performance comparable to or even better than other methods with only 100 NFEs.

### Conclusion

The paper successfully combines the generative ability of diffusion models with the traditional PnP - IR method by proposing the DiffPIR method and solves multiple challenges in image restoration. The experimental results prove that DiffPIR performs excellently in multiple tasks, especially in dealing with ill - posed inverse problems, and can efficiently generate high - quality images.