What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?

Rui Yang,Yong Lin,Xiaoteng Ma,Hao Hu,Chongjie Zhang,Tong Zhang

2023-06-02

Abstract:Offline goal-conditioned RL (GCRL) offers a way to train general-purpose agents from fully offline datasets. In addition to being conservative within the dataset, the generalization ability to achieve unseen goals is another fundamental challenge for offline GCRL. However, to the best of our knowledge, this problem has not been well studied yet. In this paper, we study out-of-distribution (OOD) generalization of offline GCRL both theoretically and empirically to identify factors that are important. In a number of experiments, we observe that weighted imitation learning enjoys better generalization than pessimism-based offline RL method. Based on this insight, we derive a theory for OOD generalization, which characterizes several important design choices. We then propose a new offline GCRL method, Generalizable Offline goAl-condiTioned RL (GOAT), by combining the findings from our theoretical and empirical studies. On a new benchmark containing 9 independent identically distributed (IID) tasks and 17 OOD tasks, GOAT outperforms current state-of-the-art methods by a large margin.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The paper primarily aims to address a key issue in Offline Goal-conditioned Reinforcement Learning (GCRL): how to enable models to generalize to Out-of-Distribution (OOD) goals. Specifically, the research focuses on how to ensure that the learned policy not only performs well on goals within the dataset but also generalizes effectively to goals outside the dataset, using only a static dataset for training.

To achieve this, the paper first conducts a series of experiments to observe the performance of several existing methods and finds that pessimism-based offline reinforcement learning methods (such as CQL+HER) and simple imitation learning methods have limitations when dealing with OOD goals. Based on these observations, the paper proposes a new theoretical framework to analyze and identify factors crucial for OOD generalization and designs a new offline GCRL algorithm accordingly—Generalizable Offline Goal-conditioned RL (GOAT).

The GOAT algorithm combines techniques such as weighted imitation learning, advantage reweighting, data selection, density reweighting, and goal relabeling, aiming to minimize the generalization bound. The paper also develops a robotic manipulation benchmark environment with multiple tasks to evaluate the performance of different algorithms on OOD generalization. Experiments demonstrate that GOAT significantly outperforms current state-of-the-art methods in this benchmark, achieving higher success rates on many OOD tasks.

In summary, the core issue addressed by this paper is the OOD generalization problem in offline goal-conditioned reinforcement learning, and it proposes an effective solution with the GOAT algorithm.