Abstract:Modern diffusion models, particularly those utilizing a Transformer-based UNet for denoising, rely heavily on self-attention operations to manage complex spatial relationships, thus achieving impressive generation performance. However, this existing paradigm faces significant challenges in generating high-resolution visual content due to its quadratic time and memory complexity with respect to the number of spatial tokens. To address this limitation, we aim at a novel linear attention mechanism as an alternative in this paper. Specifically, we begin our exploration from recently introduced models with linear complexity, e.g., Mamba2, RWKV6, Gated Linear Attention, etc, and identify two key features--attention normalization and non-causal inference--that enhance high-resolution visual generation performance. Building on these insights, we introduce a generalized linear attention paradigm, which serves as a low-rank approximation of a wide spectrum of popular linear token mixers. To save the training cost and better leverage pre-trained models, we initialize our models and distill the knowledge from pre-trained StableDiffusion (SD). We find that the distilled model, termed LinFusion, achieves performance on par with or superior to the original SD after only modest training, while significantly reducing time and memory complexity. Extensive experiments on SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion enables satisfactory and efficient zero-shot cross-resolution generation, accommodating ultra-resolution images like 16K on a single GPU. Moreover, it is highly compatible with pre-trained SD components and pipelines, such as ControlNet, IP-Adapter, DemoFusion, DistriFusion, etc, requiring no adaptation efforts. Codes are available at <a class="link-external link-https" href="https://github.com/Huage001/LinFusion" rel="external noopener nofollow">this https URL</a>.

DemoFusion: Democratising High-Resolution Image Generation With No $$$

High-Resolution Image Synthesis with Latent Diffusion Models

DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidance

Phoenix: A Federated Generative Diffusion Model

LinFusion: 1 GPU, 1 Minute, 16K Image

MegaFusion: Extend Diffusion Models towards Higher-resolution Image Generation without Further Tuning

AP-LDM: Attentive and Progressive Latent Diffusion Model for Training-Free High-Resolution Image Generation

PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher

High-Resolution Image Editing via Multi-Stage Blended Diffusion

An Acceleration Framework for High Resolution Image Synthesis

Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation

DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion

Accelerating Score-based Generative Models for High-Resolution Image Synthesis

DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis

GH-DDM: the generalized hybrid denoising diffusion model for medical image generation

DIAGen: Diverse Image Augmentation with Generative Models

Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer

HoloFusion: Towards Photo-realistic 3D Generative Modeling

DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image Generation using Limited Data

Matryoshka Diffusion Models

Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Models