Abstract:Recent advances in generative modeling with diffusion processes (DPs) enabled breakthroughs in image synthesis. Despite impressive image quality, these models have various prompt compliance problems, including low recall in generating multiple objects, difficulty in generating text in images, and meeting constraints like object locations and pose. For fine-grained editing and manipulation, they also require fine-grained semantic or instance maps that are tedious to produce manually. While prompt compliance can be enhanced by addition of loss functions at inference, this is time consuming and does not scale to complex scenes. To overcome these limitations, this work introduces a new family of \textit{Factor Graph Diffusion Models} (FG-DMs) that models the joint distribution of images and conditioning variables, such as semantic, sketch, depth or normal maps via a factor graph decomposition. This joint structure has several advantages, including support for efficient sampling based prompt compliance schemes, which produce images of high object recall, semi-automated fine-grained editing, text-based editing of conditions with noise inversion, explainability at intermediate levels, ability to produce labeled datasets for the training of downstream models such as segmentation or depth, training with missing data, and continual learning where new conditioning variables can be added with minimal or no modifications to the existing structure. We propose an implementation of FG-DMs by adapting a pre-trained Stable Diffusion (SD) model to implement all FG-DM factors, using only COCO dataset, and show that it is effective in generating images with 15\% higher recall than SD while retaining its generalization ability. We introduce an attention distillation loss that encourages consistency among the attention maps of all factors, improving the fidelity of the generated conditions and image.

Counting Guidance for High Fidelity Text-to-Image Synthesis

Make It Count: Text-to-Image Generation with an Accurate Number of Objects

Iterative Object Count Optimization for Text-to-image Diffusion Models

Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering

Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

ECNet: Effective Controllable Text-to-Image Diffusion Models

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Semantic Guidance Tuning for Text-To-Image Diffusion Models

Prompt-Free Diffusion: Taking "text" out of Text-to-Image Diffusion Models

Dense Text-to-Image Generation with Attention Modulation

SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis

Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models

Mask-ControlNet: Higher-Quality Image Generation with An Additional Mask Prompt

Segmentation-Free Guidance for Text-to-Image Diffusion Models

Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis

Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation

Improving Text-guided Object Inpainting with Semantic Pre-inpainting

Detector Guidance for Multi-Object Text-to-Image Generation

Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models

From Text to Pose to Image: Improving Diffusion Model Control and Quality