Abstract:Modern diffusion models, particularly those utilizing a Transformer-based UNet for denoising, rely heavily on self-attention operations to manage complex spatial relationships, thus achieving impressive generation performance. However, this existing paradigm faces significant challenges in generating high-resolution visual content due to its quadratic time and memory complexity with respect to the number of spatial tokens. To address this limitation, we aim at a novel linear attention mechanism as an alternative in this paper. Specifically, we begin our exploration from recently introduced models with linear complexity, e.g., Mamba2, RWKV6, Gated Linear Attention, etc, and identify two key features--attention normalization and non-causal inference--that enhance high-resolution visual generation performance. Building on these insights, we introduce a generalized linear attention paradigm, which serves as a low-rank approximation of a wide spectrum of popular linear token mixers. To save the training cost and better leverage pre-trained models, we initialize our models and distill the knowledge from pre-trained StableDiffusion (SD). We find that the distilled model, termed LinFusion, achieves performance on par with or superior to the original SD after only modest training, while significantly reducing time and memory complexity. Extensive experiments on SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion enables satisfactory and efficient zero-shot cross-resolution generation, accommodating ultra-resolution images like 16K on a single GPU. Moreover, it is highly compatible with pre-trained SD components and pipelines, such as ControlNet, IP-Adapter, DemoFusion, DistriFusion, etc, requiring no adaptation efforts. Codes are available at <a class="link-external link-https" href="https://github.com/Huage001/LinFusion" rel="external noopener nofollow">this https URL</a>.

SinFusion: Training Diffusion Models on a Single Image or Video

Video Diffusion Models

SinDiffusion: Learning a Diffusion Model from a Single Natural Image

One Diffusion to Generate Them All

ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation

Accelerating Video Diffusion Models via Distribution Matching

SF-V: Single Forward Video Generation Model

VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation

Training-free Long Video Generation with Chain of Diffusion Model Experts

Imagen Video: High Definition Video Generation with Diffusion Models

LinFusion: 1 GPU, 1 Minute, 16K Image

Photorealistic Video Generation with Diffusion Models

From Slow Bidirectional to Fast Causal Video Generators

BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models

SinDDM: A Single Image Denoising Diffusion Model

InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing

Infusion: internal diffusion for inpainting of dynamic textures and complex motion

Latent Video Diffusion Models for High-Fidelity Long Video Generation

TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generation