Abstract:Recent advances in learning decision-making policies can largely be attributed to training expressive policy models, largely via imitation learning. While imitation learning discards non-expert data, reinforcement learning (RL) can still learn from suboptimal data. However, instantiating RL training of a new policy class often presents a different challenge: most deep RL machinery is co-developed with assumptions on the policy class and backbone, resulting in poor performance when the policy class changes. For instance, SAC utilizes a low-variance reparameterization policy gradient for Gaussian policies, but this is unstable for diffusion policies and intractable for autoregressive categorical policies. To address this issue, we develop an offline RL and online fine-tuning approach called policy-agnostic RL (PA-RL) that can effectively train multiple policy classes, with varying architectures and sizes. We build off the basic idea that a universal supervised learning loss can replace the policy improvement step in RL, as long as it is applied on "optimized" actions. To obtain these optimized actions, we first sample multiple actions from a base policy, and run global optimization (i.e., re-ranking multiple action samples using the Q-function) and local optimization (i.e., running gradient steps on an action sample) to maximize the critic on these candidates. PA-RL enables fine-tuning diffusion and transformer policies with either autoregressive tokens or continuous action outputs, at different sizes, entirely via actor-critic RL. Moreover, PA-RL improves the performance and sample-efficiency by up to 2 times compared to existing offline RL and online fine-tuning methods. We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm, improving from 40% to 70% in the real world in 40 minutes.

Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

OSCAR: OOD State-Conservative Offline Reinforcement Learning for Sequential Decision Making

Cost-aware Offline Safe Meta Reinforcement Learning with Robust In-Distribution Online Task Adaptation.

Constraints Penalized Q-learning for Safe Offline Reinforcement Learning.

An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning

Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Constrained Decision Transformer for Offline Safe Reinforcement Learning

Offline Safe Reinforcement Learning Using Trajectory Classification

SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning

FOSP: Fine-tuning Offline Safe Policy through World Models

Robust Offline Reinforcement Learning from Low-Quality Data

POCE: Primal Policy Optimization with Conservative Estimation for Multi-constraint Offline Reinforcement Learning

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation

Offline Reinforcement Learning from Datasets with Structured Non-Stationarity

Mildly Constrained Evaluation Policy for Offline Reinforcement Learning

Sparsity-based Safety Conservatism for Constrained Offline Reinforcement Learning

Efficient Online Reinforcement Learning with Offline Data