Abstract:Recent advances in generative modeling with diffusion processes (DPs) enabled breakthroughs in image synthesis. Despite impressive image quality, these models have various prompt compliance problems, including low recall in generating multiple objects, difficulty in generating text in images, and meeting constraints like object locations and pose. For fine-grained editing and manipulation, they also require fine-grained semantic or instance maps that are tedious to produce manually. While prompt compliance can be enhanced by addition of loss functions at inference, this is time consuming and does not scale to complex scenes. To overcome these limitations, this work introduces a new family of \textit{Factor Graph Diffusion Models} (FG-DMs) that models the joint distribution of images and conditioning variables, such as semantic, sketch, depth or normal maps via a factor graph decomposition. This joint structure has several advantages, including support for efficient sampling based prompt compliance schemes, which produce images of high object recall, semi-automated fine-grained editing, text-based editing of conditions with noise inversion, explainability at intermediate levels, ability to produce labeled datasets for the training of downstream models such as segmentation or depth, training with missing data, and continual learning where new conditioning variables can be added with minimal or no modifications to the existing structure. We propose an implementation of FG-DMs by adapting a pre-trained Stable Diffusion (SD) model to implement all FG-DM factors, using only COCO dataset, and show that it is effective in generating images with 15\% higher recall than SD while retaining its generalization ability. We introduce an attention distillation loss that encourages consistency among the attention maps of all factors, improving the fidelity of the generated conditions and image.

FilterPrompt: A Simple yet Efficient Approach to Guide Image Appearance Transfer in Diffusion Models

Prompt-Free Diffusion: Taking "text" out of Text-to-Image Diffusion Models

ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models

Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models

IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

Manipulating Embeddings of Stable Diffusion Prompts

Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models

BudgetFusion: Perceptually-Guided Adaptive Diffusion Models

PromptFix: You Prompt and We Fix the Photo

Dynamic Prompt Optimizing for Text-to-Image Generation

Not All Steps Are Created Equal: Selective Diffusion Distillation for Image Manipulation

Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusion

Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models

Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis

HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Prompts

Prompt Diffusion Robustifies Any-Modality Prompt Learning

On Discrete Prompt Optimization for Diffusion Models

Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models

RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidance

In-Context Learning Unlocked for Diffusion Models

Diffusion Cocktail: Mixing Domain-Specific Diffusion Models for Diversified Image Generations