CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation

Kangfu Mei,Mauricio Delbracio,Hossein Talebi,Zhengzhong Tu,Vishal M. Patel,Peyman Milanfar

2024-02-17

Abstract:Large generative diffusion models have revolutionized text-to-image generation and offer immense potential for conditional generation tasks such as image enhancement, restoration, editing, and compositing. However, their widespread adoption is hindered by the high computational cost, which limits their real-time application. To address this challenge, we introduce a novel method dubbed CoDi, that adapts a pre-trained latent diffusion model to accept additional image conditioning inputs while significantly reducing the sampling steps required to achieve high-quality results. Our method can leverage architectures such as ControlNet to incorporate conditioning inputs without compromising the model's prior knowledge gained during large scale pre-training. Additionally, a conditional consistency loss enforces consistent predictions across diffusion steps, effectively compelling the model to generate high-quality images with conditions in a few steps. Our conditional-task learning and distillation approach outperforms previous distillation methods, achieving a new state-of-the-art in producing high-quality images with very few steps (e.g., 1-4) across multiple tasks, including super-resolution, text-guided image editing, and depth-to-image generation.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve

The paper aims to address the high computational cost of generative diffusion models (such as text-to-image generation) in conditional generation tasks (e.g., image enhancement, restoration, editing, and synthesis). While large generative diffusion models have made revolutionary progress in text-to-image generation and hold great potential, their widespread application is limited by high computational costs, making real-time applications difficult to achieve.

To tackle this challenge, the paper proposes a new method—CoDi (Conditional Diffusion Distillation), which can adapt pre-trained latent diffusion models to additional image condition inputs while significantly reducing the sampling steps, thereby quickly generating high-quality images. Compared to existing distillation methods, CoDi achieves significant performance improvements in various tasks (including super-resolution, text-guided image editing, and depth-to-image generation) and can generate high-quality images with only a few sampling steps (e.g., 1-4 steps).

Additionally, the CoDi method introduces a parameter-efficient distillation mechanism (PE-CoDi), which can adapt to new conditional data by adding a small number of additional parameters without sacrificing the generative performance of the pre-trained model, thereby enhancing the practicality of different conditional tasks.