Abstract:Reinforcement learning from human feedback (RLHF) methods are emerging as a way to fine-tune diffusion models (DMs) for visual generation. However, commonly used on-policy strategies are limited by the generalization capability of the reward model, while off-policy approaches require large amounts of difficult-to-obtain paired human-annotated data, particularly in visual generation tasks. To address the limitations of both on- and off-policy RLHF, we propose a preference optimization method that aligns DMs with preferences without relying on reward models or paired human-annotated data. Specifically, we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO leverages previous checkpoints as reference models while using them to generate on-policy reference samples, which replace "losing images" in preference pairs. This approach allows us to optimize using only off-policy "winning images." Furthermore, we design a strategy for reference model selection that expands the exploration in the policy space. Notably, we do not simply treat reference samples as negative examples for learning. Instead, we design an anchor-based criterion to assess whether the reference samples are likely to be winning or losing images, allowing the model to selectively learn from the generated reference samples. This approach mitigates performance degradation caused by the uncertainty in reference sample quality. We validate SePPO across both text-to-image and text-to-video benchmarks. SePPO surpasses all previous approaches on the text-to-image benchmarks and also demonstrates outstanding performance on the text-to-video benchmarks. Code will be released in <a class="link-external link-https" href="https://github.com/DwanZhang-AI/SePPO" rel="external noopener nofollow">this https URL</a>.

Sample Efficient Reinforcement Learning with Double Importance Sampling Weight Clipping

An Off-Policy Trust Region Policy Optimization Method with Monotonic Improvement Guarantee for Deep Reinforcement Learning

Generalized Proximal Policy Optimization with Sample Reuse

Optimizing DDPM Sampling with Shortcut Fine-Tuning

AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control

Policy Gradient with Active Importance Sampling

Clipped-Objective Policy Gradients for Pessimistic Policy Optimization

Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation

Proximal Policy Optimization Algorithms

Regularly Updated Deterministic Policy Gradient Algorithm

On the Reuse Bias in Off-Policy Reinforcement Learning

A dynamical clipping approach with task feedback for Proximal Policy Optimization

Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning

Authentic Boundary Proximal Policy Optimization

SePPO: Semi-Policy Preference Optimization for Diffusion Alignment

Fast Proximal Policy Optimization

Proximal Policy Optimization Smoothed Algorithm

Low-Switching Policy Gradient with Exploration via Online Sensitivity Sampling

Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations

Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning

Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling