Prototype Clustered Diffusion Models for Versatile Inverse Problems

Jinghao Zhang,Zizheng Yang,Qi Zhu,Feng Zhao

2024-07-13

Abstract:Diffusion models have made remarkable progress in solving various inverse problems, attributing to the generative modeling capability of the data manifold. Posterior sampling from the conditional score function enable the precious data consistency certified by the measurement-based likelihood term. However, most prevailing approaches confined to the deterministic deterioration process of the measurement model, regardless of capricious unpredictable disturbance in real-world sceneries. To address this obstacle, we show that the measurement-based likelihood can be renovated with restoration-based likelihood via the opposite probabilistic graphic direction, licencing the patronage of various off-the-shelf restoration models and extending the strictly deterministic deterioration process to adaptable clustered processes with the supposed prototype, in what we call restorer guidance. Particularly, assembled with versatile prototypes optionally, we can resolve inverse problems with bunch of choices for assorted sample quality and realize the proficient deterioration control with assured realistic. We show that our work can be formally analogous to the transition from classifier guidance to classifier-free guidance in the field of inverse problem solver. Experiments on multifarious inverse problems demonstrate the effectiveness of our method, including image dehazing, rain streak removal, and motion deblurring.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that when dealing with various inverse problems, the existing diffusion model methods are limited by the deterministic degradation process and cannot well adapt to the complex, changeable and unpredictable interference factors in real - world scenarios. Specifically, most of the existing methods rely on the deterministic parameters in the measurement model, which makes them insufficient when dealing with complex degradation processes in practical scenarios such as weather changes or irregular human - made interference. To overcome this obstacle, the author proposes a new method - restoration - based likelihood. By using the probability graph direction opposite to that of the traditional measurement model to improve the data consistency term, it allows the use of various off - the - shelf restoration models and extends the strictly deterministic degradation process to a highly adaptable clustering process, called "restorer guidance". This method not only improves the model's selectivity for different sample qualities but also enables effective control of the degradation process, ensuring the authenticity of the results.

In short, the paper aims to enhance the flexibility and adaptability of the diffusion model in dealing with inverse problems with unpredictable interference by introducing the restorer - guidance method.