Abstract:Goal-conditioned reinforcement learning (GCRL) refers to learning general-purpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks without additional interactions with the environment. Although offline GCRL has become increasingly prevalent and many previous works have demonstrated its empirical success, the theoretical understanding of efficient offline GCRL algorithms is not well established, especially when the state space is huge and the offline dataset only covers the policy we aim to learn. In this paper, we provide a rigorous theoretical analysis of an existing empirically successful offline GCRL algorithm. We prove that under slight modification, this algorithm enjoys an $\widetilde{O}(\text{poly}(1/\epsilon))$ sample complexity (where $\epsilon$ is the desired suboptimality of the learned policy) with general function approximation thanks to the property of (semi-)strong convexity of the objective functions. We only require nearly minimal assumptions on the dataset (single-policy concentrability) and the function class (realizability). Moreover, this algorithm consists of two uninterleaved optimization steps, which we refer to as $V$-learning and policy learning, and is computationally stable since it does not involve minimax optimization. We also empirically validate our theory by showing that the modified algorithm outperforms the previous algorithm in various real-world environments. To the best of our knowledge, this is the first algorithm that is both provably efficient with general function approximation and single-policy concentrability, and empirically successful without requiring solving minimax optimization problems.

Goal-conditioned offline reinforcement learning through state space partitioning

Goal-conditioned Offline Reinforcement Learning through State Space Partitioning

Beyond Reward: Offline Preference-guided Policy Optimization

State Advantage Weighting for Offline RL

HIQL: Offline Goal-Conditioned RL with Latent States as Actions

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective

GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models

Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning

Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability

Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

Distance Weighted Supervised Learning for Offline Interaction Data

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

Curriculum Goal-Conditioned Imitation for Offline Reinforcement Learning

Reward-free Offline Reinforcement Learning

Policy-regularized Offline Multi-objective Reinforcement Learning

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policy

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Constraints Penalized Q-learning for Safe Offline Reinforcement Learning.

Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation

State-Constrained Offline Reinforcement Learning