Abstract:Offline reinforcement learning (RL) leverages pre-collected datasets to train optimal policies. Diffusion Q-Learning (DQL), introducing diffusion models as a powerful and expressive policy class, significantly boosts the performance of offline RL. However, its reliance on iterative denoising sampling to generate actions slows down both training and inference. While several recent attempts have tried to accelerate diffusion-QL, the improvement in training and/or inference speed often results in degraded performance. In this paper, we introduce a dual policy approach, Diffusion Trusted Q-Learning (DTQL), which comprises a diffusion policy for pure behavior cloning and a practical one-step policy. We bridge the two polices by a newly introduced diffusion trust region loss. The diffusion policy maintains expressiveness, while the trust region loss directs the one-step policy to explore freely and seek modes within the region defined by the diffusion policy. DTQL eliminates the need for iterative denoising sampling during both training and inference, making it remarkably computationally efficient. We evaluate its effectiveness and algorithmic characteristics against popular Kullback--Leibler divergence-based distillation methods in 2D bandit scenarios and gym tasks. We then show that DTQL could not only outperform other methods on the majority of the D4RL benchmark tasks but also demonstrate efficiency in training and inference speeds. The PyTorch implementation is available at <a class="link-external link-https" href="https://github.com/TianyuCodings/Diffusion_Trusted_Q_Learning" rel="external noopener nofollow">this https URL</a>.

Diffusion Spectral Representation for Reinforcement Learning

Policy Representation via Diffusion Probability Model for Reinforcement Learning

Diffusion Models for Reinforcement Learning: A Survey

Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning

Sampling from Energy-based Policies using Diffusion

Diffusion Policies creating a Trust Region for Offline Reinforcement Learning

Large-scale Reinforcement Learning for Diffusion Models

Training Diffusion Models with Reinforcement Learning

Diffusion Reward: Learning Rewards via Conditional Video Diffusion

Reward Shaping via Diffusion Process in Reinforcement Learning

Reinforcement Learning for Jump-Diffusions, with Financial Applications

Towards Controllable Diffusion Models via Reward-Guided Exploration

Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models

Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient

Efficient Diffusion Policies for Offline Reinforcement Learning

A reinforcement learning diffusion decision model for value-based decisions

Feedback Efficient Online Fine-Tuning of Diffusion Models

Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning

Diffusion Models as Optimizers for Efficient Planning in Offline RL

Diffusion Actor-Critic with Entropy Regulator