Abstract:Recent advances in learning decision-making policies can largely be attributed to training expressive policy models, largely via imitation learning. While imitation learning discards non-expert data, reinforcement learning (RL) can still learn from suboptimal data. However, instantiating RL training of a new policy class often presents a different challenge: most deep RL machinery is co-developed with assumptions on the policy class and backbone, resulting in poor performance when the policy class changes. For instance, SAC utilizes a low-variance reparameterization policy gradient for Gaussian policies, but this is unstable for diffusion policies and intractable for autoregressive categorical policies. To address this issue, we develop an offline RL and online fine-tuning approach called policy-agnostic RL (PA-RL) that can effectively train multiple policy classes, with varying architectures and sizes. We build off the basic idea that a universal supervised learning loss can replace the policy improvement step in RL, as long as it is applied on "optimized" actions. To obtain these optimized actions, we first sample multiple actions from a base policy, and run global optimization (i.e., re-ranking multiple action samples using the Q-function) and local optimization (i.e., running gradient steps on an action sample) to maximize the critic on these candidates. PA-RL enables fine-tuning diffusion and transformer policies with either autoregressive tokens or continuous action outputs, at different sizes, entirely via actor-critic RL. Moreover, PA-RL improves the performance and sample-efficiency by up to 2 times compared to existing offline RL and online fine-tuning methods. We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm, improving from 40% to 70% in the real world in 40 minutes.

Theoretical foundations for programmatic reinforcement learning

Human-Readable Programs as Actors of Reinforcement Learning Agents Using Critic-Moderated Evolution

Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs

Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search

Topological Foundations of Reinforcement Learning

Challenges of real-world reinforcement learning: definitions, benchmarks and analysis

Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges

Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization

Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

Submodular Reinforcement Learning

Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networks

Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

Generalized Reinforcement Learning: Experience Particles, Action Operator, Reinforcement Field, Memory Association, and Decision Concepts

Reinforcement Learning for Economic Policy: A New Frontier?

The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions

Counterfactual Explanation Policies in RL

Interpretable Policies for Reinforcement Learning by Genetic Programming

Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards

Programmatic Reward Design by Example

Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

Policy Rehearsing: Training Generalizable Policies for Reinforcement Learning