Abstract:Diffusion models are the current state of the art for generating photorealistic images. Controlling the sampling process for constrained image generation tasks such as inpainting, however, remains challenging since exact conditioning on such constraints is intractable. While existing methods use various techniques to approximate the constrained posterior, this paper proposes to exploit the ability of Tractable Probabilistic Models (TPMs) to exactly and efficiently compute the constrained posterior, and to leverage this signal to steer the denoising process of diffusion models. Specifically, this paper adopts a class of expressive TPMs termed Probabilistic Circuits (PCs). Building upon prior advances, we further scale up PCs and make them capable of guiding the image generation process of diffusion models. Empirical results suggest that our approach can consistently improve the overall quality and semantic coherence of inpainted images across three natural image datasets (i.e., CelebA-HQ, ImageNet, and LSUN) with only ~10% additional computational overhead brought by the TPM. Further, with the help of an image encoder and decoder, our method can readily accept semantic constraints on specific regions of the image, which opens up the potential for more controlled image generation tasks. In addition to proposing a new framework for constrained image generation, this paper highlights the benefit of more tractable models and motivates the development of expressive TPMs.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in constrained image generation tasks such as inpainting, how to effectively guide diffusion models to generate high - quality images that meet the constraints. Specifically, the paper proposes solutions to the following challenges: 1. **Limitations of existing methods**: - Although diffusion models perform well in generating realistic images, they have difficulties when dealing with constrained image generation tasks, such as inpainting. Because these models cannot efficiently calculate the posterior sample distribution with constraints. - Existing methods solve this problem by approximately calculating the posterior distribution with constraints, but these approximation methods introduce high biases, which affect the quality of the generated images. 2. **Introduction of Tractable Probabilistic Models (TPMs)**: - To overcome the above problems, the paper proposes to use a class of tractable probabilistic models (TPMs) called Probabilistic Circuits (PCs). These models can efficiently and accurately calculate the posterior distribution with constraints. - Specifically, the paper shows how to use PCs to guide the denoising process of diffusion models, thereby generating high - quality images that are more in line with the constraints. 3. **Improving the quality of inpainting**: - The method proposed in the paper can significantly improve the overall quality and semantic consistency of the inpainted images on three natural image datasets (CelebA - HQ, ImageNet, and LSUN), while only increasing the computational overhead by about 10%. - In addition, by combining image encoders and decoders, this method can accept semantic constraints on specific regions of the image, further enhancing the controllability of image generation. In summary, this paper aims to solve the difficult problem of generating high - quality images that meet the constraints in inpainting tasks by combining diffusion models and tractable probabilistic models, and provides a new framework and solution for this purpose.

Image Inpainting via Tractable Steering of Diffusion Models

Coherent and Multi-modality Image Inpainting via Latent Space Optimization

TD-Paint: Faster Diffusion Inpainting Through Time Aware Pixel Conditioning

Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis

Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models

Progressive Inpainting Strategy with Partial Convolutions Generative Networks (PPCGN).

A Diffusion Model with A FFT for Image Inpainting

3D-Consistent Image Inpainting with Diffusion Models

GradPaint: Gradient-guided inpainting with diffusion models

Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model

VIPaint: Image Inpainting with Pre-Trained Diffusion Models via Variational Inference

Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images

Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors

Denoising Diffusion Probabilistic Models for Image Inpainting of Cell Distributions in the Human Brain

Controlled and Conditional Text to Image Generation with Diffusion Prior

Not All Steps Are Created Equal: Selective Diffusion Distillation for Image Manipulation

Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits

High-Resolution Image Synthesis with Latent Diffusion Models

Sketch-guided Image Inpainting with Partial Discrete Diffusion Process

Fast constrained sampling in pre-trained diffusion models

G2D2: Gradient-guided Discrete Diffusion for image inverse problem solving