Abstract:Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality. It has been verified that utilizing diffusion policies can significantly improve the performance of RL algorithms in continuous control tasks by overcoming the limitations of unimodal policies, such as Gaussian policies, and providing the agent with enhanced exploration capabilities. However, existing works mainly focus on the application of diffusion policies in offline RL, while their incorporation into online RL is less investigated. The training objective of the diffusion model, known as the variational lower bound, cannot be optimized directly in online RL due to the unavailability of 'good' actions. This leads to difficulties in conducting diffusion policy improvement. To overcome this, we propose a novel model-free diffusion-based online RL algorithm, Q-weighted Variational Policy Optimization (QVPO). Specifically, we introduce the Q-weighted variational loss, which can be proved to be a tight lower bound of the policy objective in online RL under certain conditions. To fulfill these conditions, the Q-weight transformation functions are introduced for general scenarios. Additionally, to further enhance the exploration capability of the diffusion policy, we design a special entropy regularization term. We also develop an efficient behavior policy to enhance sample efficiency by reducing the variance of the diffusion policy during online interactions. Consequently, the QVPO algorithm leverages the exploration capabilities and multimodality of diffusion policies, preventing the RL agent from converging to a sub-optimal policy. To verify the effectiveness of QVPO, we conduct comprehensive experiments on MuJoCo benchmarks. The final results demonstrate that QVPO achieves state-of-the-art performance on both cumulative reward and sample efficiency.

Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning

Efficient Diffusion Policies for Offline Reinforcement Learning

Diffusion Policies creating a Trust Region for Offline Reinforcement Learning

Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning

Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization

DiffPoGAN: Diffusion Policies with Generative Adversarial Networks for Offline Reinforcement Learning

Reasoning with Latent Diffusion in Offline Reinforcement Learning

Score Regularized Policy Optimization Through Diffusion Behavior

Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning

DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning

Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning

Offline Reinforcement Learning with On-Policy Q-Function Regularization

Policy Representation via Diffusion Probability Model for Reinforcement Learning

Offline Reinforcement Learning with Reverse Diffusion Guide Policy

Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning

Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

Hypercube Policy Regularization Framework for Offline Reinforcement Learning

Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models

Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control