Abstract:Recent advances in learning decision-making policies can largely be attributed to training expressive policy models, largely via imitation learning. While imitation learning discards non-expert data, reinforcement learning (RL) can still learn from suboptimal data. However, instantiating RL training of a new policy class often presents a different challenge: most deep RL machinery is co-developed with assumptions on the policy class and backbone, resulting in poor performance when the policy class changes. For instance, SAC utilizes a low-variance reparameterization policy gradient for Gaussian policies, but this is unstable for diffusion policies and intractable for autoregressive categorical policies. To address this issue, we develop an offline RL and online fine-tuning approach called policy-agnostic RL (PA-RL) that can effectively train multiple policy classes, with varying architectures and sizes. We build off the basic idea that a universal supervised learning loss can replace the policy improvement step in RL, as long as it is applied on "optimized" actions. To obtain these optimized actions, we first sample multiple actions from a base policy, and run global optimization (i.e., re-ranking multiple action samples using the Q-function) and local optimization (i.e., running gradient steps on an action sample) to maximize the critic on these candidates. PA-RL enables fine-tuning diffusion and transformer policies with either autoregressive tokens or continuous action outputs, at different sizes, entirely via actor-critic RL. Moreover, PA-RL improves the performance and sample-efficiency by up to 2 times compared to existing offline RL and online fine-tuning methods. We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm, improving from 40% to 70% in the real world in 40 minutes.

Phasic Parallel-Network Policy: a Deep Reinforcement Learning Framework Based on Action Correlation

Beyond Reward: Offline Preference-guided Policy Optimization

Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimization

Approximate Policy-Based Accelerated Deep Reinforcement Learning.

Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

Data Efficient Training for Reinforcement Learning with Adaptive Behavior Policy Sharing

Efficient Deep Reinforcement Learning Through Policy Transfer.

Unified Policy Optimization for Continuous-action Reinforcement Learning in Non-stationary Tasks and Games

Actor-Critic Reinforcement Learning with Phased Actor

Scalable Model-based Policy Optimization for Decentralized Networked Systems

Network Architecture for Optimizing Deep Deterministic Policy Gradient Algorithms

Efficient Deep Reinforcement Learning Via Adaptive Policy Transfer

Multi-Path Policy Optimization

OMPO: A Unified Framework for RL under Policy and Dynamics Shifts

Adversarial Policy Optimization in Deep Reinforcement Learning

Efficient Parallel Methods for Deep Reinforcement Learning

Multiple Suboptimal Policies Integrated Reinforcement Learning Algorithm for Path Planning

Deep reinforcement learning algorithm based on multi-agent parallelism and its application in game environment

PP-PG: Combining Parameter Perturbation with Policy Gradient Methods for Effective and Efficient Explorations in Deep Reinforcement Learning

Decentralized Policy Optimization

Population-Guided Parallel Policy Search for Reinforcement Learning