Zhiyuan Ma,Yuzhu Zhang,Guoli Jia,Liangliang Zhao,Yichao Ma,Mingjie Ma,Gaofeng Liu,Kaiyan Zhang,Jianjun Li,Bowen Zhou
Abstract:As one of the most popular and sought-after generative models in the recent years, diffusion models have sparked the interests of many researchers and steadily shown excellent advantage in various generative tasks such as image synthesis, video generation, molecule design, 3D scene rendering and multimodal generation, relying on their dense theoretical principles and reliable application practices. The remarkable success of these recent efforts on diffusion models comes largely from progressive design principles and efficient architecture, training, inference, and deployment methodologies. However, there has not been a comprehensive and in-depth review to summarize these principles and practices to help the rapid understanding and application of diffusion models. In this survey, we provide a new efficiency-oriented perspective on these existing efforts, which mainly focuses on the profound principles and efficient practices in architecture designs, model training, fast inference and reliable deployment, to guide further theoretical research, algorithm migration and model application for new scenarios in a reader-friendly way. \url{<a class="link-external link-https" href="https://github.com/ponyzym/Efficient-DMs-Survey" rel="external noopener nofollow">this https URL</a>}
What problem does this paper attempt to address?
The paper attempts to address the shortcomings of current Diffusion Models (DMs) in terms of efficiency, scalability, and practical applications. Although diffusion models have shown outstanding performance in tasks such as image synthesis, video generation, molecular design, 3D scene rendering, and multimodal generation, there is a lack of a comprehensive and in-depth review to summarize the design principles, training methods, fast inference, and reliable deployment practices of these models. This survey aims to systematically organize the existing research progress from the perspective of efficiency, covering aspects such as architecture design, model training, fast inference, and reliable deployment, to guide future theoretical research, algorithm migration, and model applications in new scenarios.
Specifically, the paper focuses on the following issues:
1. **Theoretical Foundations**: Explain and reveal the essential reasons for the generation effects of diffusion models, sorting out related theories such as dynamic modeling, score matching, latent projection, and conditional guidance, to promote the development of new theories and guide various efficient generation practices.
2. **Efficient Architectures**: Explore mainstream diffusion model backbone networks, such as U-Net, DiT, U-ViT, MamBa, etc., analyze their design structures, compare their respective advantages and disadvantages, to guide the emergence of more powerful new deep scalable architectures.
3. **Efficient Training and Fine-tuning**: Organize efficient training, fine-tuning, and preference optimization methods for diffusion models, such as low-rank adaptation, consistency training, adversarial training, adapter training, etc., to help researchers and developers make appropriate choices in specific low-resource or personalized training tasks.
4. **Efficient Sampling and Inference**: Investigate the most commonly used efficient sampling and inference strategies in diffusion models, including non-learning and learning-based methods, by comparing their acceleration performance in various generation tasks, providing a theoretical basis for the research of faster sampling methods.
5. **Efficient Deployment**: Summarize the latest deployment solutions of current diffusion models on mobile devices and web pages, promoting operations in various cross-platform, low-resource environments, and driving the birth of various applications.
6. **Practical Applications**: Discuss the practical applications of efficient diffusion models in various fields, emphasizing the balance between generation performance, efficiency, and computational cost.
By discussing these issues, the paper hopes to provide readers with a comprehensive understanding of the current state-of-the-art efficient generation models and to point out directions for future research and applications, promoting a deeper understanding of the challenges and opportunities in the field of efficient diffusion models.