Abstract:Diffusion Probabilistic Models stand as a critical tool in generative modelling, enabling the generation of complex data distributions. This family of generative models yields record-breaking performance in tasks such as image synthesis, video generation, and molecule design. Despite their capabilities, their efficiency, especially in the reverse process, remains a challenge due to slow convergence rates and high computational costs. In this paper, we introduce an approach that leverages continuous dynamical systems to design a novel denoising network for diffusion models that is more parameter-efficient, exhibits faster convergence, and demonstrates increased noise robustness. Experimenting with Denoising Diffusion Probabilistic Models (DDPMs), our framework operates with approximately a quarter of the parameters, and $\sim$ 30\% of the Floating Point Operations (FLOPs) compared to standard U-Nets in DDPMs. Furthermore, our model is notably faster in inference than the baseline when measured in fair and equal conditions. We also provide a mathematical intuition as to why our proposed reverse process is faster as well as a mathematical discussion of the empirical tradeoffs in the denoising downstream task. Finally, we argue that our method is compatible with existing performance enhancement techniques, enabling further improvements in efficiency, quality, and speed.

What problem does this paper attempt to address?

This paper focuses on improving the efficiency of Diffusion Probabilistic Models (DPMMs), especially in their reverse process. DPMMs play an important role in generative modeling and are applicable to generating complex data distributions such as image synthesis, video generation, and molecular design. Despite their excellent performance, the slow convergence speed and high computational cost in the reverse process remain a challenge. The authors propose a new approach that utilizes continuous dynamical systems to design a novel denoising network, which improves parameter efficiency, convergence speed, and noise robustness. Compared to the standard U-Net used in traditional DPMMs, their framework uses approximately one-quarter of the parameters and about 30% of the floating-point operations (FLOPs). Moreover, the proposed model also significantly outperforms the baseline model in inference speed under fair and equivalent conditions. In this research, they develop a new denoising diffusion probabilistic model (DDPM) with a denoising network based on a continuous deep latent U-Net, which replaces the traditional discrete U-Net. This approach improves efficiency by reducing memory usage, increasing inference speed, and maintaining image quality. The paper also discusses the potential of combining this approach with other performance-enhancing techniques to further improve efficiency, quality, and speed. Experimental results show that the proposed framework performs comparably to the standard methods in image synthesis tasks, but with improvements in inference time and denoising performance in the reverse process. The paper also provides mathematical intuitions and discusses the trade-offs made in the experiments, demonstrating the compatibility of the proposed method with existing acceleration strategies.

The Missing U for Efficient Diffusion Models

Denoising Diffusion Straightforward Models for Energy Conversion Monitoring Data Imputation

Denoising Diffusion Step-aware Models

UDPM: Upsampling Diffusion Probabilistic Models

Accelerating Diffusion Models via Early Stop of the Diffusion Process

Efficient Denoising Diffusion Via Probabilistic Masking

Diffusion Models Without Attention

Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models

Improving Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architectures

Interpreting and Improving Diffusion Models from an Optimization Perspective

Learning to Discretize Denoising Diffusion ODEs

Q-DM: an Efficient Low-bit Quantized Diffusion Model

Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference

Think While You Generate: Discrete Diffusion with Planned Denoising

DuoDiff: Accelerating Diffusion Models with a Dual-Backbone Approach

Filling Deterministic Approximate Gap in Denoising Diffusion Probabilistic Models with Energy

Fast Inference in Denoising Diffusion Models via MMD Finetuning

Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection

AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision