Abstract:We present SDXL, a latent diffusion model for text-to-image synthesis. Compared to previous versions of Stable Diffusion, SDXL leverages a three times larger UNet backbone: The increase of model parameters is mainly due to more attention blocks and a larger cross-attention context as SDXL uses a second text encoder. We design multiple novel conditioning schemes and train SDXL on multiple aspect ratios. We also introduce a refinement model which is used to improve the visual fidelity of samples generated by SDXL using a post-hoc image-to-image technique. We demonstrate that SDXL shows drastically improved performance compared the previous versions of Stable Diffusion and achieves results competitive with those of black-box state-of-the-art image generators. In the spirit of promoting open research and fostering transparency in large model training and evaluation, we provide access to code and model weights at <a class="link-external link-https" href="https://github.com/Stability-AI/generative-models" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on improving the performance of text - to - image synthesis models, especially in high - resolution image generation. Specifically, the paper proposes SDXL (Stable Diffusion eXtended Large), which is an improved version of the latent diffusion model, aiming to solve the following key problems: 1. **Increasing the number of model parameters and the complexity of the architecture**: - Compared with previous versions of Stable Diffusion, SDXL uses a UNet backbone network three times larger, adding more attention blocks and a larger cross - attention context. This is mainly achieved by introducing a second text encoder. - Formula representation: \[ \text{UNet number of parameters} = 2.6B \] 2. **Multi - condition scheme**: - Designed a variety of new condition schemes that do not require additional supervision. For example, the model can be conditioned on image size and cropping parameters to better handle images of different sizes and proportions. - Image size conditioning: \[ c_{\text{size}}=(h_{\text{original}}, w_{\text{original}}) \] - Cropping parameter conditioning: \[ c_{\text{crop}}=(c_{\text{top}}, c_{\text{left}}) \] 3. **Multi - aspect - ratio training**: - Train the model to handle images with multiple aspect ratios, thereby improving the versatility and adaptability of the model. - By dividing the data set into buckets with different aspect ratios and training within each bucket, ensure that the model can handle images of various proportions. 4. **Refinement model**: - Introduced a refinement model to improve the visual fidelity of samples generated by SDXL. This refinement model uses post - processing image - to - image techniques to improve image quality through a noise - denoising process. - The noise - denoising process of the refinement model can be represented by the following formula: \[ x_{\text{refined}}=\text{Denoise}(x_{\text{noisy}}) \] 5. **Improving the quality of generated images**: - Through the above improvements, SDXL significantly outperforms previous versions of Stable Diffusion in generating high - quality images and is comparable in performance to the state - of - the - art image - generation models in the black - box state. 6. **Open research and transparency**: - In order to promote open research and improve the transparency of large - model training and evaluation, the authors provide access to the code and model weights. In summary, through a series of technological innovations and improvements, this paper aims to improve the performance of text - to - image synthesis models, especially for high - resolution image generation, while maintaining the openness and transparency of the model.

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis

Progressive Knowledge Distillation Of Stable Diffusion XL Using Layer Level Loss

Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets

SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions

UniFL: Improve Latent Diffusion Model via Unified Feedback Learning

Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation

Text-driven Visual Synthesis with Latent Diffusion Prior

Aligning Diffusion Models with Noise-Conditioned Perception

SDXL-Lightning: Progressive Adversarial Diffusion Distillation

High-Resolution Image Synthesis with Latent Diffusion Models

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Large-scale Reinforcement Learning for Diffusion Models

Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis

Improving Text Generation on Images with Synthetic Captions

CommonCanvas: An Open Diffusion Model Trained with Creative-Commons Images

SpeedUpNet: A Plug-and-Play Hyper-Network for Accelerating Text-to-Image Diffusion Models

KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis

BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion

ControlNet-XS: Designing an Efficient and Effective Architecture for Controlling Text-to-Image Diffusion Models