Abstract:Standard reinforcement learning (RL) agents never intelligently explore like a human (i.e. taking into account complex domain priors and adapting quickly based on previous exploration). Across episodes, RL agents struggle to perform even simple exploration strategies, for example systematic search that avoids exploring the same location multiple times. This poor exploration limits performance on challenging domains. Meta-RL is a potential solution, as unlike standard RL, meta-RL can learn to explore, and potentially learn highly complex strategies far beyond those of standard RL, strategies such as experimenting in early episodes to learn new skills, or conducting experiments to learn about the current environment. Traditional meta-RL focuses on the problem of learning to optimally balance exploration and exploitation to maximize the cumulative reward of the episode sequence (e.g., aiming to maximize the total wins in a tournament -- while also improving as a player). We identify a new challenge with state-of-the-art cumulative-reward meta-RL methods. When optimal behavior requires exploration that sacrifices immediate reward to enable higher subsequent reward, existing state-of-the-art cumulative-reward meta-RL methods become stuck on the local optimum of failing to explore. Our method, First-Explore, overcomes this limitation by learning two policies: one to solely explore, and one to solely exploit. When exploring requires forgoing early-episode reward, First-Explore significantly outperforms existing cumulative meta-RL methods. By identifying and solving the previously unrecognized problem of forgoing reward in early episodes, First-Explore represents a significant step towards developing meta-RL algorithms capable of human-like exploration on a broader range of domains.

RL$^3$: Boosting Meta Reinforcement Learning via RL inside RL$^2$

Meta-Reinforcement Learning Robust to Distributional Shift Via Performing Lifelong In-Context Learning

A Survey of Meta-Reinforcement Learning

Model-based Adversarial Meta-Reinforcement Learning

Introducing Symmetries to Black Box Meta Reinforcement Learning

AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers

Unsupervised Meta-Learning for Reinforcement Learning

Guided Meta-Policy Search

Meta-Reinforcement Learning with Dynamic Adaptiveness Distillation

First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs

Curriculum in Gradient-Based Meta-Reinforcement Learning

Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learning

Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approach

MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning

Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator

Meta Reinforcement Learning with Task Embedding and Shared Policy.

Multi-objective Meta-return Reinforcement Learning for Sequential Recommendation

Meta-Learning Integration in Hierarchical Reinforcement Learning for Advanced Task Complexity

Black box meta-learning intrinsic rewards for sparse-reward environments

MGHRL: Meta Goal-generation for Hierarchical Reinforcement Learning

Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery