Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Tianhe Yu,Deirdre Quillen,Zhanpeng He,Ryan Julian,Avnish Narayan,Hayden Shively,Adithya Bellathur,Karol Hausman,Chelsea Finn,Sergey Levine

DOI: https://doi.org/10.48550/arXiv.1910.10897

2021-06-15

Abstract:Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks. Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 7 state-of-the-art meta-reinforcement learning and multi-task learning algorithms on these tasks. Surprisingly, while each task and its variations (e.g., with different object positions) can be learned with reasonable success, these algorithms struggle to learn with multiple tasks at the same time, even with as few as ten distinct training tasks. Our analysis and open-source environments pave the way for future research in multi-task learning and meta-learning that can enable meaningful generalization, thereby unlocking the full potential of these methods.

Machine Learning,Artificial Intelligence,Robotics

What problem does this paper attempt to address?

The problem this paper attempts to address is the limitations in evaluation and benchmarking of current Meta-Reinforcement Learning (Meta-RL) and Multi-Task Reinforcement Learning (Multi-Task RL) methods. Specifically:

1. **Narrow Task Distribution**: Existing Meta-RL research mainly focuses on very narrow task distributions, such as robot running tasks at different speeds. This narrow task distribution makes the learned strategies unable to generalize to entirely new tasks.
2. **Lack of Shared Structure**: Multi-Task RL methods are usually evaluated on unrelated tasks, such as Atari games, which lack shared structure between tasks, making it difficult to improve the performance of other tasks by learning one task.
3. **Insufficient Generalization Ability**: Although some multi-task and meta-learning methods perform well on single tasks, their performance significantly drops when dealing with a diverse set of tasks, and they even fail to generalize to entirely new tasks.

To overcome these issues, the paper proposes a new benchmarking platform—Meta-World, which includes 50 different robotic manipulation tasks. These tasks are not only diverse but also have shared structures, aiming to evaluate and advance the development of multi-task and meta-reinforcement learning algorithms, enabling them to generalize more effectively to entirely new tasks.