Abstract:Goal-conditioned reinforcement learning (GCRL) refers to learning general-purpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks without additional interactions with the environment. Although offline GCRL has become increasingly prevalent and many previous works have demonstrated its empirical success, the theoretical understanding of efficient offline GCRL algorithms is not well established, especially when the state space is huge and the offline dataset only covers the policy we aim to learn. In this paper, we provide a rigorous theoretical analysis of an existing empirically successful offline GCRL algorithm. We prove that under slight modification, this algorithm enjoys an $\widetilde{O}(\text{poly}(1/\epsilon))$ sample complexity (where $\epsilon$ is the desired suboptimality of the learned policy) with general function approximation thanks to the property of (semi-)strong convexity of the objective functions. We only require nearly minimal assumptions on the dataset (single-policy concentrability) and the function class (realizability). Moreover, this algorithm consists of two uninterleaved optimization steps, which we refer to as $V$-learning and policy learning, and is computationally stable since it does not involve minimax optimization. We also empirically validate our theory by showing that the modified algorithm outperforms the previous algorithm in various real-world environments. To the best of our knowledge, this is the first algorithm that is both provably efficient with general function approximation and single-policy concentrability, and empirically successful without requiring solving minimax optimization problems.

Curriculum Goal-Conditioned Imitation for Offline Reinforcement Learning

Beyond Reward: Offline Preference-guided Policy Optimization

Curriculum Offline Imitation Learning

Curriculum offline imitating learning

Curriculum Offline Reinforcement Learning

Are Expressive Models Truly Necessary for Offline RL?

GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models

Goal-conditioned offline reinforcement learning through state space partitioning

Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability

Self-Supervised Imitation for Offline Reinforcement Learning with Hindsight Relabeling

Using Offline Data to Speed-up Reinforcement Learning in Procedurally Generated Environments

HIQL: Offline Goal-Conditioned RL with Latent States as Actions

GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation

Goal-Conditioned Predictive Coding for Offline Reinforcement Learning

When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

Curricular Subgoals for Inverse Reinforcement Learning