Abstract:We propose WSAC (Weighted Safe Actor-Critic), a novel algorithm for Safe Offline Reinforcement Learning (RL) under functional approximation, which can robustly optimize policies to improve upon an arbitrary reference policy with limited data coverage. WSAC is designed as a two-player Stackelberg game to optimize a refined objective function. The actor optimizes the policy against two adversarially trained value critics with small importance-weighted Bellman errors, which focus on scenarios where the actor's performance is inferior to the reference policy. In theory, we demonstrate that when the actor employs a no-regret optimization oracle, WSAC achieves a number of guarantees: (i) For the first time in the safe offline RL setting, we establish that WSAC can produce a policy that outperforms any reference policy while maintaining the same level of safety, which is critical to designing a safe algorithm for offline RL. (ii) WSAC achieves the optimal statistical convergence rate of $1/\sqrt{N}$ to the reference policy, where $N$ is the size of the offline dataset. (iii) We theoretically show that WSAC guarantees a safe policy improvement across a broad range of hyperparameters that control the degree of pessimism, indicating its practical robustness. Additionally, we offer a practical version of WSAC and compare it with existing state-of-the-art safe offline RL algorithms in several continuous control environments. WSAC outperforms all baselines across a range of tasks, supporting the theoretical results.

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

Behavior Proximal Policy Optimization

Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Double Actors and Uncertainty-Weighted Critics for Offline Reinforcement Learning.

Dual Behavior Regularized Offline Deterministic Actor–Critic

Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

Actor-Critic Alignment for Offline-to-Online Reinforcement Learning

In-sample Actor Critic for Offline Reinforcement Learning

Robust Offline Actor-Critic with On-Policy Regularized Policy Evaluation

Efficient Offline Reinforcement Learning: The Critic is Critical

CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

Offline Reinforcement Learning With Behavior Value Regularization

Robust Offline Reinforcement Learning from Low-Quality Data

Risk-Averse Offline Reinforcement Learning

Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning

Offline Meta-Reinforcement Learning with Advantage Weighting

ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

Toward a data efficient neural actor-critic