Hierarchical Integration Diffusion Model for Realistic Image Deblurring

Zheng Chen,Yulun Zhang,Ding Liu,Bin Xia,Jinjin Gu,Linghe Kong,Xin Yuan

2023-09-26

Abstract:Diffusion models (DMs) have recently been introduced in image deblurring and exhibited promising performance, particularly in terms of details reconstruction. However, the diffusion model requires a large number of inference iterations to recover the clean image from pure Gaussian noise, which consumes massive computational resources. Moreover, the distribution synthesized by the diffusion model is often misaligned with the target results, leading to restrictions in distortion-based metrics. To address the above issues, we propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Specifically, we perform the DM in a highly compacted latent space to generate the prior feature for the deblurring process. The deblurring process is implemented by a regression-based method to obtain better distortion accuracy. Meanwhile, the highly compact latent space ensures the efficiency of the DM. Furthermore, we design the hierarchical integration module to fuse the prior into the regression-based model from multiple scales, enabling better generalization in complex blurry scenarios. Comprehensive experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods. Code and trained models are available at <a class="link-external link-https" href="https://github.com/zhengchen1999/HI-Diff" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address several key challenges in Image Deblurring:

1. **High Computational Cost**: Existing Diffusion Models (DMs) require numerous inference iterations to generate clear images, consuming significant computational resources.
2. **Distribution Inconsistency**: The distribution generated by diffusion models often does not align with the target result, leading to poor performance on distortion-based evaluation metrics such as PSNR.
3. **Insufficient Detail Recovery**: Traditional regression methods are conservative in recovering high-frequency details, while generative models (e.g., GANs) can generate complex details but are prone to artifacts not present in the original clear image.

To tackle these challenges, the paper proposes a new method—Hierarchical Integration Diffusion Model (HI-Diff), aimed at achieving more efficient image deblurring with better generalization in complex blurry scenarios.

### Main Contributions:
1. **Efficient Generation of Prior Features**: HI-Diff applies diffusion models in a highly compressed latent space to generate prior features, significantly reducing computational complexity.
2. **Hierarchical Integration Module**: A Hierarchical Integration Module (HIM) is designed to fuse multi-scale prior features into the regression model, enhancing detail recovery quality and generalization ability.
3. **Two-Stage Training Strategy**: A two-stage training strategy is adopted, involving latent compression and diffusion model training separately, ensuring the model's effectiveness and stability.

### Method Overview:
1. **Stage 1: Latent Compression**:
- A Latent Encoder (LE) is used to compress real images into highly compact latent representations as prior features.
- The prior features are integrated into the Transformer through the Hierarchical Integration Module (HIM) to guide the deblurring process.

2. **Stage 2: Latent Diffusion Model**:
- A Latent Diffusion Model (DM) is trained to generate prior features, enhancing the Transformer's deblurring performance through HIM.
- A joint training strategy is employed to ensure collaborative optimization between the diffusion model and the Transformer.

### Experimental Results:
- **Ablation Studies**: The effectiveness of diffusion priors, hierarchical integration, and joint training strategies is validated through comparative experiments with different designs.
- **Benchmark Testing**: Extensive experiments on synthetic datasets (GoPro, HIDE) and real-world datasets (RealBlur, RWBI) show that HI-Diff outperforms existing state-of-the-art methods on multiple evaluation metrics (e.g., PSNR, SSIM).

In summary, the paper successfully addresses the issues of high computational cost and insufficient detail recovery in image deblurring through innovative design and effective training strategies, providing a more efficient and robust solution for practical applications.