Abstract:Optimizing a text-to-image diffusion model with a given reward function is an important but underexplored research area. In this study, we propose Deep Reward Tuning (DRTune), an algorithm that directly supervises the final output image of a text-to-image diffusion model and back-propagates through the iterative sampling process to the input noise. We find that training earlier steps in the sampling process is crucial for low-level rewards, and deep supervision can be achieved efficiently and effectively by stopping the gradient of the denoising network input. DRTune is extensively evaluated on various reward models. It consistently outperforms other algorithms, particularly for low-level control signals, where all shallow supervision methods fail. Additionally, we fine-tune Stable Diffusion XL 1.0 (SDXL 1.0) model via DRTune to optimize Human Preference Score v2.1, resulting in the Favorable Diffusion XL 1.0 (FDXL 1.0) model. FDXL 1.0 significantly enhances image quality compared to SDXL 1.0 and reaches comparable quality compared with Midjourney v5.2.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to optimize the text - to - image diffusion model given the reward function. Specifically, existing methods face several challenges when optimizing the diffusion model through the reward model: 1. **Computational Overhead**: Directly back - propagating gradients from the output image to the input noise will lead to significant memory consumption, especially when dealing with text - to - image diffusion models with multiple sampling steps. 2. **Gradient Explosion**: The problem of gradient explosion is prone to occur during the early sampling steps of training, which will hinder the convergence of the model. 3. **Low - level Reward Optimization**: Existing methods perform poorly when optimizing certain low - level rewards (such as symmetry rewards). To solve these problems, the paper proposes the **Deep Reward Tuning (DRTune)** algorithm, which efficiently supervises the early denoising steps of the diffusion model through the following two main improvements: 1. **Stop - gradient**: Stop the gradient at the input of the denoising network, thereby avoiding the gradient explosion problem and accelerating convergence. 2. **Selective Training Sampling Steps**: In each training iteration, a subset is uniformly spaced selected from all sampling steps for back - propagation, so that the untrained intermediate steps can be skipped, reducing the computational and memory overhead. Through these improvements, DRTune can optimize the diffusion model more effectively, especially performing well when optimizing low - level rewards. The paper also shows the use of DRTune to fine - tune Stable Diffusion XL 1.0 (SDXL 1.0) to optimize the Human Preference Score v2.1 (HPS v2.1), generating the Favorable Diffusion XL 1.0 (FDXL 1.0) model. Experimental results show that FDXL 1.0 is significantly superior to SDXL 1.0 in image quality and is comparable to the commercial model Midjourney v5.2.5.

Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

Directly Fine-Tuning Diffusion Models on Differentiable Rewards

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation

PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models

Enhancing Diffusion Models with Text-Encoder Reinforcement Learning

Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward

Reward Incremental Learning in Text-to-Image Generation

Aligning Text-to-Image Diffusion Models with Reward Backpropagation

Large-scale Reinforcement Learning for Diffusion Models

InstructVideo: Instructing Video Diffusion Models with Human Feedback

Diffusion Model Alignment Using Direct Preference Optimization

Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control

RFSR: Improving ISR Diffusion Models via Reward Feedback Learning

ReDiFine: Reusable Diffusion Finetuning for Mitigating Degradation in the Chain of Diffusion

Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models

Training Diffusion Models with Reinforcement Learning

Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models

Feedback Efficient Online Fine-Tuning of Diffusion Models