Abstract:Although the diffusion model has achieved remarkable performance in the field of image generation, its high inference delay hinders its wide application in edge devices with scarce computing resources. Therefore, many training-free sampling methods have been proposed to reduce the number of sampling steps required for diffusion models. However, they perform poorly under a very small number of sampling steps. Thanks to the emergence of knowledge distillation technology, the existing training scheme methods have achieved excellent results at very low step numbers. However, the current methods mainly focus on designing novel diffusion model sampling methods with knowledge distillation. How to transfer better diffusion knowledge from teacher models is a more valuable problem but rarely studied. Therefore, we propose Relational Diffusion Distillation (RDD), a novel distillation method tailored specifically for distilling diffusion models. Unlike existing methods that simply align teacher and student models at pixel level or feature distributions, our method introduces cross-sample relationship interaction during the distillation process and alleviates the memory constraints induced by multiple sample interactions. Our RDD significantly enhances the effectiveness of the progressive distillation framework within the diffusion model. Extensive experiments on several datasets (e.g., CIFAR-10 and ImageNet) demonstrate that our proposed RDD leads to 1.47 FID decrease under 1 sampling step compared to state-of-the-art diffusion distillation methods and achieving 256x speed-up compared to DDIM strategy. Code is available at <a class="link-external link-https" href="https://github.com/cantbebetter2/RDD" rel="external noopener nofollow">this https URL</a>.

Incremental Image Generation with Diffusion Models by Label Embedding Initialization and Fusion

Few-Shot Incremental Learning for Label-to-Image Translation

Data-Free Federated Class Incremental Learning with Diffusion-Based Generative Memory

FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classes

An Efficient Framework for Enhancing Discriminative Models via Diffusion Techniques

Diffusion Models Are Innate One-Step Generators

DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation

Diffusion Models Need Visual Priors for Image Generation

Relational Diffusion Distillation for Efficient Image Generation

Learning on Less: Constraining Pre-trained Model Learning for Generalizable Diffusion-Generated Image Detection

Masked Diffusion Models Are Fast Distribution Learners

Diffusion Cocktail: Mixing Domain-Specific Diffusion Models for Diversified Image Generations

Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion

Future-Proofing Class Incremental Learning

Diffusion Cocktail: Fused Generation from Diffusion Models

Accelerated Image-Aware Generative Diffusion Modeling

One Diffusion to Generate Them All

Adaptively Controllable Diffusion Model for Efficient Conditional Image Generation

Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?

Latent-based Diffusion Model for Long-tailed Recognition