Abstract:Meta-reinforcement learning (meta-RL) acquires meta-policies that show good performance for tasks in a wide task distribution. However, conventional meta-RL, which learns meta-policies by randomly sampling tasks, has been reported to show meta-overfitting for certain tasks, especially for easy tasks where an agent can easily get high scores. To reduce effects of the meta-overfitting, we considered meta-RL with curriculum-based task sampling. Our method is Robust Meta Reinforcement Learning with Guided Task Sampling (RMRL-GTS), which is an effective method that restricts task sampling based on scores and epochs. We show that in order to achieve robust meta-RL, it is necessary not only to intensively sample tasks with poor scores, but also to restrict and expand the task regions of the tasks to be sampled.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the meta - overfitting problem that occurs in traditional meta - reinforcement learning (meta - RL) methods when dealing with a wide range of task distributions. Specifically: 1. **Meta - overfitting problem**: Traditional meta - reinforcement learning methods, such as MAML (Model - Agnostic Meta - Learning), learn meta - policies by randomly sampling tasks from the task distribution. However, this method is prone to meta - overfitting, especially on easy tasks where agents can easily obtain high scores, thus ignoring more difficult tasks. 2. **Insufficient ability to adapt to complex tasks**: Due to the meta - overfitting phenomenon, meta - policies are difficult to adapt to difficult tasks from a wide range of task distributions. This limits the robustness and generalization ability of meta - reinforcement learning in practical applications. To solve these problems, the authors propose a task - sampling method based on curriculum learning - **Robust Meta - Reinforcement Learning with Guided Task Sampling (RMRL - GTS)**. This method aims to reduce the impact of meta - overfitting and improve the adaptability of meta - policies to a wide range of task distributions through the following two strategies: 1. **Limitation of task - sampling areas**: Dynamically adjust the task - sampling area according to task difficulty and training rounds. In the initial stage, focus on simpler tasks and gradually expand to more difficult tasks as training progresses. 2. **Score - based priority sampling**: Give priority to sampling tasks with lower scores (i.e., more difficult tasks) to ensure that these tasks have sufficient learning opportunities. Through these two strategies, RMRL - GTS can pay more balanced attention to tasks of different difficulties during the training process, thereby improving the overall robustness of meta - policies and better adapting to tasks from a wide range of task distributions. ### Formula summary - **MAML parameter update formula**: \[ \theta'_i=\theta - \alpha\nabla_\theta L_{\tau_i}(\pi_\theta) \] where \(L_{\tau_i}\) is the loss function of task \(\tau_i\) and \(\alpha\) is the step size. - **Meta - parameter update formula**: \[ \theta=\theta - \beta\nabla_\theta\sum_{\tau_i\sim p(\tau)} L_{\tau_i}(\pi'_{\theta_i}) \] where \(\beta\) is the step size. - **Weighted average total reward**: \[ f(\tau)=\frac{1}{n_{\text{bin}}}\sum_{\tau < \tau' < \tau + d\tau_{\text{bin}}}\sum_{i = 0}^{n_{\text{ce}}} c_i(\tau')R_{\text{mean},i}(\tau') \] where \(c_i(\tau)\) is the weight and \(R_{\text{mean},i}(\tau)\) is the average total reward after the \(i\)-th round of training. - **Probability calculation formula**: \[ p(\tau)=\frac{1-\bar{f}(\tau)}{\sum_{\tau'}(1 - \bar{f}(\tau'))} \] where \(\bar{f}(\tau)\) is the normalized interpolation function. Through these methods, RMRL - GTS can effectively reduce the meta - overfitting phenomenon and improve the robustness and generalization ability of meta - reinforcement learning.

Robust Meta-Reinforcement Learning with Curriculum-Based Task Sampling

Meta-Reinforcement Learning Robust to Distributional Shift Via Performing Lifelong In-Context Learning

Curriculum in Gradient-Based Meta-Reinforcement Learning

Guided Meta-Policy Search

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Model-based Adversarial Meta-Reinforcement Learning

Meta-Reinforcement Learning Based on Self-Supervised Task Representation Learning

Constrained Meta Agnostic Reinforcement Learning

A Survey of Meta-Reinforcement Learning

Importance Sampling-Guided Meta-Training for Intelligent Agents in Highly Interactive Environments

Prediction Guided Meta-Learning for Multi-Objective Reinforcement Learning

Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator

Exploration With Task Information for Meta Reinforcement Learning

MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration

Meta Reinforcement Learning with Finite Training Tasks -- a Density Estimation Approach

Robust Driving Policy Learning with Guided Meta Reinforcement Learning

Context-Based Meta-Reinforcement Learning With Bayesian Nonparametric Models

Meta-Reinforcement Learning in Nonstationary and Nonparametric Environments

Towards Task Sampler Learning for Meta-Learning

Meta-Gradient Search Control: A Method for Improving the Efficiency of Dyna-style Planning