Abstract:Deep reinforcement learning agents usually need to collect a large number of interactions to solve a single task. In contrast, meta-reinforcement learning (meta-RL) aims to quickly adapt to new tasks using a small amount of experience by leveraging the knowledge from training on a set of similar tasks. State-of-the-art context-based meta-RL algorithms use the context to encode the task information and train a policy conditioned on the inferred latent task encoding. However, most recent works are limited to parametric tasks, where a handful of variables control the full variation in the task distribution, and also failed to work in non-stationary environments due to the few-shot adaptation setting. To address those limitations, we propose MEta-reinforcement Learning with Task Self-discovery (MELTS), which adaptively learns qualitatively different nonparametric tasks and adapts to new tasks in a zero-shot manner. We introduce a novel deep clustering framework (DPMM-VAE) based on an infinite mixture of Gaussians, which combines the Dirichlet process mixture model (DPMM) and the variational autoencoder (VAE), to simultaneously learn task representations and cluster the tasks in a self-adaptive way. Integrating DPMM-VAE into MELTS enables it to adaptively discover the multi-modal structure of the nonparametric task distribution, which previous methods using isotropic Gaussian random variables cannot model. In addition, we propose a zero-shot adaptation mechanism and a recurrence-based context encoding strategy to improve the data efficiency and make our algorithm applicable in non-stationary environments. On various continuous control tasks with both parametric and nonparametric variations, our algorithm produces a more structured and self-adaptive task latent space and also achieves superior sample efficiency and asymptotic performance compared with state-of-the-art meta-RL algorithms.

Robust Meta-Reinforcement Learning with Curriculum-Based Task Sampling

Meta-Reinforcement Learning Robust to Distributional Shift Via Performing Lifelong In-Context Learning

Curriculum in Gradient-Based Meta-Reinforcement Learning

Guided Meta-Policy Search

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Model-based Adversarial Meta-Reinforcement Learning

Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning

Constrained Meta Agnostic Reinforcement Learning

A Survey of Meta-Reinforcement Learning

Importance Sampling-Guided Meta-Training for Intelligent Agents in Highly Interactive Environments

Prediction Guided Meta-Learning for Multi-Objective Reinforcement Learning

Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator

Exploration With Task Information for Meta Reinforcement Learning

MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration

Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approach

Robust Driving Policy Learning with Guided Meta Reinforcement Learning

Context-Based Meta-Reinforcement Learning With Bayesian Nonparametric Models

Meta-Reinforcement Learning in Nonstationary and Nonparametric Environments

Towards Task Sampler Learning for Meta-Learning

Meta-Gradient Search Control: A Method for Improving the Efficiency of Dyna-style Planning