Abstract:Existing offline reinforcement learning (RL) methods face a few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL is emerging as a promising approach to address these challenges, aiming to learn an informative meta-policy from a collection of tasks. Nevertheless, as shown in our empirical studies, offline Meta-RL could be outperformed by offline single-task RL methods on tasks with good quality of datasets, indicating that a right balance has to be delicately calibrated between"exploring"the out-of-distribution state-actions by following the meta-policy and"exploiting"the offline dataset by staying close to the behavior policy. Motivated by such empirical analysis, we explore model-based offline Meta-RL with regularized Policy Optimization (MerPO), which learns a meta-model for efficient task structure inference and an informative meta-policy for safe exploration of out-of-distribution state-actions. In particular, we devise a new meta-Regularized model-based Actor-Critic (RAC) method for within-task policy optimization, as a key building block of MerPO, using conservative policy evaluation and regularized policy improvement; and the intrinsic tradeoff therein is achieved via striking the right balance between two regularizers, one based on the behavior policy and the other on the meta-policy. We theoretically show that the learnt policy offers guaranteed improvement over both the behavior policy and the meta-policy, thus ensuring the performance improvement on new tasks via offline Meta-RL. Experiments corroborate the superior performance of MerPO over existing offline Meta-RL methods.

MER: Modular Element Randomization for Robust Generalizable Policy in Deep Reinforcement Learning

Zero-Shot Policy Transfer with Disentangled Task Representation of Meta-Reinforcement Learning.

Learning Invariable Semantical Representation from Language for Extensible Policy Generalization

Non-local Policy Optimization via Diversity-regularized Collaborative Exploration

RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning

Heterogeneous Knowledge for Augmented Modular Reinforcement Learning

A Game-Theoretic Perspective of Generalization in Reinforcement Learning

Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble.

Policy Rehearsing: Training Generalizable Policies for Reinforcement Learning

$\epsilon$-Invariant Hierarchical Reinforcement Learning for Building Generalizable Policy

Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms

Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution

Model-Based Offline Meta-Reinforcement Learning with Regularization

Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs

Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation

AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning

Mixture of Experts in a Mixture of RL settings

Cooperative Modular Reinforcement Learning for Large Discrete Action Space Problem.

Low-Rank Modular Reinforcement Learning via Muscle Synergy

Multi-Task Reinforcement Learning with Soft Modularization.