Abstract:Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality. It has been verified that utilizing diffusion policies can significantly improve the performance of RL algorithms in continuous control tasks by overcoming the limitations of unimodal policies, such as Gaussian policies, and providing the agent with enhanced exploration capabilities. However, existing works mainly focus on the application of diffusion policies in offline RL, while their incorporation into online RL is less investigated. The training objective of the diffusion model, known as the variational lower bound, cannot be optimized directly in online RL due to the unavailability of 'good' actions. This leads to difficulties in conducting diffusion policy improvement. To overcome this, we propose a novel model-free diffusion-based online RL algorithm, Q-weighted Variational Policy Optimization (QVPO). Specifically, we introduce the Q-weighted variational loss, which can be proved to be a tight lower bound of the policy objective in online RL under certain conditions. To fulfill these conditions, the Q-weight transformation functions are introduced for general scenarios. Additionally, to further enhance the exploration capability of the diffusion policy, we design a special entropy regularization term. We also develop an efficient behavior policy to enhance sample efficiency by reducing the variance of the diffusion policy during online interactions. Consequently, the QVPO algorithm leverages the exploration capabilities and multimodality of diffusion policies, preventing the RL agent from converging to a sub-optimal policy. To verify the effectiveness of QVPO, we conduct comprehensive experiments on MuJoCo benchmarks. The final results demonstrate that QVPO achieves state-of-the-art performance on both cumulative reward and sample efficiency.

QUANTILE-BASED POLICY OPTIMIZATION FOR REINFORCEMENT LEARNING

Behavior Proximal Policy Optimization

Unified Policy Optimization for Robust Reinforcement Learning

Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement Learning

Proximal Policy Optimization Algorithms

Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization

REBEL: Reinforcement Learning via Regressing Relative Rewards

Risk-Aware Constrained Reinforcement Learning with Non-Stationary Policies.

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization

Proximal Policy Optimization with Future Rewards

Policy Optimization for Continuous Reinforcement Learning

Provably Efficient Exploration in Policy Optimization

Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble.

Fully Parameterized Quantile Function for Distributional Reinforcement Learning.

Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data

Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards

Towards applicable reinforcement learning: Improving the generalization and sample efficiency with policy ensemble

POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning

Quantile Reinforcement Learning

Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning