Abstract:In this work, we decouple the iterative (bi-level) offline RL optimization from the offline training phase, forming a non-iterative bi-level learning paradigm that avoids the iterative error propagation over two levels. Specifically, this non-iterative paradigm allows us to conduct inner-level optimization in training (ie, employing policy/value regularization), while performing outer-level optimization in testing (ie, conducting policy inference). Naturally, such paradigm raises three core questions (that are not fully answered by prior non-iterative methods): (Q1) What information should we transfer from inner-level to outer-level? (Q2) What should we pay attention to when using the transferred information in outer-level optimization? (Q3) What are the benefits of concurrently conducting outer-level optimization during testing? Motivated by model-based optimization, we proposed DROP, which fully answered the above three questions. Particularly, in inner-level, DROP decomposes offline data into multiple subsets, and learns a score model (Q1). To keep safe exploitation to score model in outer-level, we explicitly learn a behavior embedding and introduce a conservative regularization (Q2). During testing, we show that DROP permits deployment adaptation, enabling an adaptive inference across states (Q3). Empirically, we evaluate DROP on various benchmarks, showing that DROP gains comparable or better performance compared to prior offline RL methods.

Actor-Critic Alignment for Offline-to-Online Reinforcement Learning

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

Behavior Proximal Policy Optimization

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

In-sample Actor Critic for Offline Reinforcement Learning

Offline–Online Actor–Critic

Efficient Offline Reinforcement Learning: The Critic is Critical

Offline-Online Actor-Critic for Partially Observable Markov Decision Process

Dual Behavior Regularized Offline Deterministic Actor–Critic

Double Actors and Uncertainty-Weighted Critics for Offline Reinforcement Learning.

Online Meta-Critic Learning for Off-Policy Actor-Critic Methods

Robust Offline Reinforcement Learning from Low-Quality Data

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

Online Tuning for Offline Decentralized Multi-Agent Reinforcement Learning

Off-Policy Neural Fitted Actor-Critic

Understanding the performance gap between online and offline alignment algorithms

Robust Offline Actor-Critic with On-Policy Regularized Policy Evaluation

Offline Reinforcement Learning With Behavior Value Regularization

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

Sample Efficient Offline-to-Online Reinforcement Learning