Abstract:Recent advances in learning decision-making policies can largely be attributed to training expressive policy models, largely via imitation learning. While imitation learning discards non-expert data, reinforcement learning (RL) can still learn from suboptimal data. However, instantiating RL training of a new policy class often presents a different challenge: most deep RL machinery is co-developed with assumptions on the policy class and backbone, resulting in poor performance when the policy class changes. For instance, SAC utilizes a low-variance reparameterization policy gradient for Gaussian policies, but this is unstable for diffusion policies and intractable for autoregressive categorical policies. To address this issue, we develop an offline RL and online fine-tuning approach called policy-agnostic RL (PA-RL) that can effectively train multiple policy classes, with varying architectures and sizes. We build off the basic idea that a universal supervised learning loss can replace the policy improvement step in RL, as long as it is applied on "optimized" actions. To obtain these optimized actions, we first sample multiple actions from a base policy, and run global optimization (i.e., re-ranking multiple action samples using the Q-function) and local optimization (i.e., running gradient steps on an action sample) to maximize the critic on these candidates. PA-RL enables fine-tuning diffusion and transformer policies with either autoregressive tokens or continuous action outputs, at different sizes, entirely via actor-critic RL. Moreover, PA-RL improves the performance and sample-efficiency by up to 2 times compared to existing offline RL and online fine-tuning methods. We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm, improving from 40% to 70% in the real world in 40 minutes.

Offline Adaptive Policy Leaning in Real-World Sequential Recommendation Systems

Beyond Reward: Offline Preference-guided Policy Optimization

Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Pseudo Dyna-Q

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning

Sim-to-Real Interactive Recommendation via Off-Dynamics Reinforcement Learning

ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems

Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systems

Towards Data-Driven Offline Simulations for Online Reinforcement Learning

Diversification of Adaptive Policy for Effective Offline Reinforcement Learning

NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning.

Reformulating Conversational Recommender Systems as Tri-Phase Offline Policy Learning

Offline Fictitious Self-Play for Competitive Games

Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation

Deploying Offline Reinforcement Learning with Human Feedback

M 3 Rec: A Context-aware Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation

PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators