Abstract:In offline reinforcement learning (RL), it is necessary to manage out-of-distribution actions to prevent overestimation of value functions. Policy-regularized methods address this problem by constraining the target policy to stay close to the behavior policy. Although several approaches suggest representing the behavior policy as an expressive diffusion model to boost performance, it remains unclear how to regularize the target policy given a diffusion-modeled behavior sampler. In this paper, we propose Diffusion Actor-Critic (DAC) that formulates the Kullback-Leibler (KL) constraint policy iteration as a diffusion noise regression problem, enabling direct representation of target policies as diffusion models. Our approach follows the actor-critic learning paradigm that we alternatively train a diffusion-modeled target policy and a critic network. The actor training loss includes a soft Q-guidance term from the Q-gradient. The soft Q-guidance grounds on the theoretical solution of the KL constraint policy iteration, which prevents the learned policy from taking out-of-distribution actions. For critic training, we train a Q-ensemble to stabilize the estimation of Q-gradient. Additionally, DAC employs lower confidence bound (LCB) to address the overestimation and underestimation of value targets due to function approximation error. Our approach is evaluated on the D4RL benchmarks and outperforms the state-of-the-art in almost all environments. Code is available at \href{<a class="link-external link-https" href="https://github.com/Fang-Lin93/DAC" rel="external noopener nofollow">this https URL</a>}{\texttt{<a class="link-external link-http" href="http://github.com/Fang-Lin93/DAC" rel="external noopener nofollow">this http URL</a>}}.

In-sample Actor Critic for Offline Reinforcement Learning

Behavior Proximal Policy Optimization

Actor-Critic Alignment for Offline-to-Online Reinforcement Learning

A Rank-Based Sampling Framework for Offline Reinforcement Learning

Offline Reinforcement Learning With Behavior Value Regularization

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

Double Actors and Uncertainty-Weighted Critics for Offline Reinforcement Learning.

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

Dual Behavior Regularized Offline Deterministic Actor–Critic

Adaptable Conservative Q-Learning for Offline Reinforcement Learning.

Offline-Online Actor-Critic for Partially Observable Markov Decision Process

Offline Reinforcement Learning with Soft Behavior Regularization

Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning

Robust Offline Actor-Critic with On-Policy Regularized Policy Evaluation

Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning

Robust Offline Reinforcement Learning from Low-Quality Data

Off-Policy Neural Fitted Actor-Critic

Offline–Online Actor–Critic

Efficient Offline Reinforcement Learning: The Critic is Critical

OSCAR: OOD State-Conservative Offline Reinforcement Learning for Sequential Decision Making

Optimal Actor-Critic Policy With Optimized Training Datasets