Abstract:Diffusion models suffer severe object repetition and local distortion when the inference resolution differs from its pre-trained resolution. We propose AccDiffusion v2, an accurate method for patch-wise higher-resolution diffusion extrapolation without training. Our in-depth analysis in this paper shows that using an identical text prompt for different patches leads to repetitive generation, while the absence of a prompt undermines image details. In response, our AccDiffusion v2 novelly decouples the vanilla image-content-aware prompt into a set of patch-content-aware prompts, each of which serves as a more precise description of a patch. Further analysis reveals that local distortion arises from inaccurate descriptions in prompts about the local structure of higher-resolution images. To address this issue, AccDiffusion v2, for the first time, introduces an auxiliary local structural information through ControlNet during higher-resolution diffusion extrapolation aiming to mitigate the local distortions. Finally, our analysis indicates that global semantic information is conducive to suppressing both repetitive generation and local distortion. Hence, our AccDiffusion v2 further proposes dilated sampling with window interaction for better global semantic information during higher-resolution diffusion extrapolation. We conduct extensive experiments, including both quantitative and qualitative comparisons, to demonstrate the efficacy of our AccDiffusion v2. The quantitative comparison shows that AccDiffusion v2 achieves state-of-the-art performance in image generation extrapolation without training. The qualitative comparison intuitively illustrates that AccDiffusion v2 effectively suppresses the issues of repetitive generation and local distortion in image generation extrapolation. Our code is available at \url{<a class="link-external link-https" href="https://github.com/lzhxmu/AccDiffusion_v2" rel="external noopener nofollow">this https URL</a>}.

Diffusion Model Patching via Mixture-of-Prompts

Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models

Prompt Diffusion Robustifies Any-Modality Prompt Learning

Mix-DDPM: Enhancing Diffusion Models Through Fitting Mixture Noise with Global Stochastic Offset

DiffNAS: Bootstrapping Diffusion Models by Prompting for Better Architectures

PFDiff: Training-free Acceleration of Diffusion Models through the Gradient Guidance of Past and Future

DiffNAS: Bootstrapping Diffusion Models by Prompting for Better Architectures

Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generation

AccDiffusion v2: Towards More Accurate Higher-Resolution Diffusion Extrapolation

Boosting Diffusion Models with an Adaptive Momentum Sampler

Partially Conditioned Patch Parallelism for Accelerated Diffusion Model Inference

Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis

AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models

Efficient Denoising Diffusion Via Probabilistic Masking

Prompt Mixing in Diffusion Models using the Black Scholes Algorithm

PromptFix: You Prompt and We Fix the Photo

CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Method

Transferring Pretrained Diffusion Probabilistic Models

Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection

Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models

BudgetFusion: Perceptually-Guided Adaptive Diffusion Models