Abstract:Gradient-based methods enable efficient search capabilities in high dimensions. However, in order to apply them effectively in offline optimization paradigms such as offline Reinforcement Learning (RL) or Imitation Learning (IL), we require a more careful consideration of how uncertainty estimation interplays with first-order methods that attempt to minimize them. We study smoothed distance to data as an uncertainty metric, and claim that it has two beneficial properties: (i) it allows gradient-based methods that attempt to minimize uncertainty to drive iterates to data as smoothing is annealed, and (ii) it facilitates analysis of model bias with Lipschitz constants. As distance to data can be expensive to compute online, we consider settings where we need amortize this computation. Instead of learning the distance however, we propose to learn its gradients directly as an oracle for first-order optimizers. We show these gradients can be efficiently learned with score-matching techniques by leveraging the equivalence between distance to data and data likelihood. Using this insight, we propose Score-Guided Planning (SGP), a planning algorithm for offline RL that utilizes score-matching to enable first-order planning in high-dimensional problems, where zeroth-order methods were unable to scale, and ensembles were unable to overcome local minima. Website: <a class="link-external link-https" href="https://sites.google.com/view/score-guided-planning/home" rel="external noopener nofollow">this https URL</a>

A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

A Rank-Based Sampling Framework for Offline Reinforcement Learning

Uncertainty-aware Distributional Offline Reinforcement Learning

UAC: Offline Reinforcement Learning with Uncertain Action Constraint

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Uncertainty-Aware Data Augmentation for Offline Reinforcement Learning

Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness

SUF: Stabilized Unconstrained Fine-Tuning for Offline-to-Online Reinforcement Learning

Unsupervised-to-Online Reinforcement Learning

Uncertainty-based Bootstrapped Optimization for Offline Reinforcement Learning

Uncertainty-Aware Model-Based Offline Reinforcement Learning for Automated Driving

Offline Reinforcement Learning with Policy Guidance and Uncertainty Estimation.

Bayesian Design Principles for Offline-to-Online Reinforcement Learning

Sample Efficient Offline-to-Online Reinforcement Learning

Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching

Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation

Offline Reinforcement Learning with Generative Adversarial Networks and Uncertainty Estimation

SAMG: State-Action-Aware Offline-to-Online Reinforcement Learning with Offline Model Guidance

Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL

SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning

SDV: Simple Double Validation Model-based Offline Reinforcement Learning