Alignment of Diffusion Models: Fundamentals, Challenges, and Future

Buhua Liu,Shitong Shao,Bao Li,Lichen Bai,Zhiqiang Xu,Haoyi Xiong,James Kwok,Sumi Helal,Zeke Xie

2024-09-12

Abstract:Diffusion models have emerged as the leading paradigm in generative modeling, excelling in various applications. Despite their success, these models often misalign with human intentions, generating outputs that may not match text prompts or possess desired properties. Inspired by the success of alignment in tuning large language models, recent studies have investigated aligning diffusion models with human expectations and preferences. This work mainly reviews alignment of diffusion models, covering advancements in fundamentals of alignment, alignment techniques of diffusion models, preference benchmarks, and evaluation for diffusion models. Moreover, we discuss key perspectives on current challenges and promising future directions on solving the remaining challenges in alignment of diffusion models. To the best of our knowledge, our work is the first comprehensive review paper for researchers and engineers to comprehend, practice, and research alignment of diffusion models.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the inconsistency between Diffusion Models and human intentions and expectations. Although Diffusion Models perform excellently in generative modeling and are widely used in image generation, video generation, text generation, and other fields, the results generated by these models often fail to fully meet human expectations or preferences. For example, the generated images may not match the text prompts or may have low aesthetic quality, and the generated molecules may lack high binding affinity and structural rationality.

To solve these problems, the paper reviews alignment techniques for Diffusion Models and explores how to better align these models with human intentions and preferences. The paper mainly covers the following aspects:

1. **Alignment Basics**: Introduces the basic concepts and techniques of alignment, including the collection and modeling of preference data, alignment algorithms, etc.
2. **Alignment Techniques for Diffusion Models**: Discusses specific alignment methods for Diffusion Models in detail, such as Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), etc.
3. **Benchmarking and Evaluation**: Introduces the benchmarks and evaluation metrics used to assess the alignment effectiveness of Diffusion Models.
4. **Future Research Directions**: Discusses the challenges faced by current alignment techniques and proposes future research directions to further improve the performance and practicality of Diffusion Models.

Through these contents, the paper provides researchers and engineers with a comprehensive understanding and practical guide to help them make progress in the alignment research of Diffusion Models.