Abstract:Although the diffusion model has achieved remarkable performance in the field of image generation, its high inference delay hinders its wide application in edge devices with scarce computing resources. Therefore, many training-free sampling methods have been proposed to reduce the number of sampling steps required for diffusion models. However, they perform poorly under a very small number of sampling steps. Thanks to the emergence of knowledge distillation technology, the existing training scheme methods have achieved excellent results at very low step numbers. However, the current methods mainly focus on designing novel diffusion model sampling methods with knowledge distillation. How to transfer better diffusion knowledge from teacher models is a more valuable problem but rarely studied. Therefore, we propose Relational Diffusion Distillation (RDD), a novel distillation method tailored specifically for distilling diffusion models. Unlike existing methods that simply align teacher and student models at pixel level or feature distributions, our method introduces cross-sample relationship interaction during the distillation process and alleviates the memory constraints induced by multiple sample interactions. Our RDD significantly enhances the effectiveness of the progressive distillation framework within the diffusion model. Extensive experiments on several datasets (e.g., CIFAR-10 and ImageNet) demonstrate that our proposed RDD leads to 1.47 FID decrease under 1 sampling step compared to state-of-the-art diffusion distillation methods and achieving 256x speed-up compared to DDIM strategy. Code is available at <a class="link-external link-https" href="https://github.com/cantbebetter2/RDD" rel="external noopener nofollow">this https URL</a>.

Imagine Flash: Accelerating Emu Diffusion Models with Backward Distillation

Flash Diffusion: Accelerating Any Conditional Diffusion Model for Few Steps Image Generation

EM Distillation for One-step Diffusion Models

Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference

Accelerated Image-Aware Generative Diffusion Modeling

AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation

Accelerating Diffusion Models with One-to-Many Knowledge Distillation

Relational Diffusion Distillation for Efficient Image Generation

Accelerating Video Diffusion Models via Distribution Matching

Accelerating Dataset Distillation Via Model Augmentation

SFDDM: Single-fold Distillation for Diffusion models

DuoDiff: Accelerating Diffusion Models with a Dual-Backbone Approach

Your Student is Better Than Expected: Adaptive Teacher-Student Collaboration for Text-Conditional Diffusion Models

Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation

Latent Dataset Distillation with Diffusion Models

Plug-and-Play Diffusion Distillation

Multi-student Diffusion Distillation for Better One-step Generators

AdaDiff: Adaptive Step Selection for Fast Diffusion.

Adv-KD: Adversarial Knowledge Distillation for Faster Diffusion Sampling

Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures

One-step Diffusion with Distribution Matching Distillation