Abstract:Offline meta-reinforcement learning (offline meta-RL) extracts knowledge from a given dataset of multiple tasks and achieves fast adaptation to new tasks. Recent offline meta-RL methods typically use task-dependent behavior policies (e.g., training RL agents on each individual task) to collect a multi-task dataset and learn an offline meta-policy. However, these methods always require extra information for fast adaptation, such as offline context for testing tasks or oracle reward functions. Offline meta-RL with few-shot online adaptation remains an open problem. In this paper, we first formally characterize a unique challenge under this setting: data distribution mismatch between offline training and online adaptation. This distribution mismatch may lead to unreliable offline policy evaluation and the regular adaptation methods of online meta-RL will suffer. To address this challenge, we introduce a novel mechanism of data distribution correction, which ensures the consistency between offline and online evaluation by filtering out out-of-distribution episodes in online adaptation. As few-shot out-of-distribution episodes usually have lower returns, we propose a Greedy Context-based data distribution Correction approach, called GCC, which greedily infers how to solve new tasks. GCC diversely samples “task hypotheses” from the current posterior belief and selects a greedy hypothesis with the highest return to update the task belief. Our method is the first to provide an effective online adaptation without additional information, and can be combined with off-the-shelf context-based offline meta-training algorithms. Empirical experiments show that GCC achieves state-of-the-art performance on the Meta-World ML1 benchmark compared to baselines with/without offline adaptation.

Offline Meta Reinforcement Learning with In-Distribution Online Adaptation

Meta-Reinforcement Learning Robust to Distributional Shift Via Performing Lifelong In-Context Learning

Correcting Data Distribution Mismatch in Offline Meta-Reinforcement Learning with Few-Shot Online Adaptation

Cost-aware Offline Safe Meta Reinforcement Learning with Robust In-Distribution Online Task Adaptation.

Offline Meta-Reinforcement Learning with Advantage Weighting

Sample Efficient Offline-to-Online Reinforcement Learning

Meta-Reinforcement Learning with Dynamic Adaptiveness Distillation

Uncertainty-aware Distributional Offline Reinforcement Learning

Offline Reinforcement Learning with Imbalanced Datasets

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Robust Offline Reinforcement Learning from Low-Quality Data

Offline Meta Learning of Exploration

On Context Distribution Shift in Task Representation Learning for Offline Meta RL

Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator

Domain Adaptation for Offline Reinforcement Learning with Limited Samples

Boosting Offline Reinforcement Learning via Data Rebalancing

Model-Based Offline Meta-Reinforcement Learning with Regularization

Augmenting Offline RL with Unlabeled Data

An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learning

Model-Based Offline Adaptive Policy Optimization with Episodic Memory