Abstract:Unsupervised goal-conditioned reinforcement learning (GCRL) is a promising paradigm for developing diverse robotic skills without external supervision. However, existing unsupervised GCRL methods often struggle to cover a wide range of states in complex environments due to their limited exploration and sparse or noisy rewards for GCRL. To overcome these challenges, we propose a novel unsupervised GCRL method that leverages TemporaL Distance-aware Representations (TLDR). TLDR selects faraway goals to initiate exploration and computes intrinsic exploration rewards and goal-reaching rewards, based on temporal distance. Specifically, our exploration policy seeks states with large temporal distances (i.e. covering a large state space), while the goal-conditioned policy learns to minimize the temporal distance to the goal (i.e. reaching the goal). Our experimental results in six simulated robotic locomotion environments demonstrate that our method significantly outperforms previous unsupervised GCRL methods in achieving a wide variety of states.

What problem does this paper attempt to address?

The paper aims to address two main challenges in Unsupervised Goal-Conditioned Reinforcement Learning (GCRL): 1) exploring diverse states to ensure the agent can learn to achieve a wide range of goals; and 2) effectively learning goal-directed policies. To overcome the difficulty of existing unsupervised GCRL methods in covering a wide range of states in complex environments, the authors propose a new unsupervised GCRL method based on Temporal Distance-Aware Representations (TLDR). This method uses temporal distance to select goals that are far from already visited states and calculates intrinsic exploration rewards and goal-reaching rewards based on temporal distance. Specifically, the exploration policy seeks states with large temporal distances (i.e., covering a larger state space), while the goal-conditioned policy learns to minimize the temporal distance to reach the goal (i.e., goal achievement). Experimental results show that in 6 simulated robotic navigation environments, this method significantly outperforms previous unsupervised GCRL methods in achieving a wide range of states. Furthermore, ablation studies comparing different exploration strategies and GCRL reward designs further validate the importance of temporal distance-aware representations in enhancing exploration capabilities and optimizing goal-directed policies. In summary, the core contribution of this paper lies in proposing a method that effectively addresses the challenges of exploration and goal-directed policy learning in unsupervised GCRL by introducing the concept of temporal distance to guide the agent's exploration behavior and policy learning process.

TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations

Learning Hierarchical Graph-Based Policy for Goal-Reaching in Unknown Environments

Goal-Conditioned Reinforcement Learning with Disentanglement-based Reachability Planning

Weakly Supervised Disentangled Representation for Goal-conditioned Reinforcement Learning

A Goal-Conditioned Reinforcement Learning Algorithm with Environment Modeling

Unsupervised Control Through Non-Parametric Discriminative Rewards

Goal exploration augmentation via pre-trained skills for sparse-reward long-horizon goal-conditioned reinforcement learning

Self-imitation guided goal-conditioned reinforcement learning

Active Hierarchical Exploration with Stable Subgoal Representation Learning

Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning

Goal-Conditioned Reinforcement Learning: Problems and Solutions

Goal Exploration via Adaptive Skill Distribution for Goal-Conditioned Reinforcement Learning

Robot Subgoal-guided Navigation in Dynamic Crowded Environments with Hierarchical Deep Reinforcement Learning

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

HIQL: Offline Goal-Conditioned RL with Latent States as Actions

On-Robot Reinforcement Learning with Goal-Contrastive Rewards

Mission-driven Exploration for Accelerated Deep Reinforcement Learning with Temporal Logic Task Specifications

Learning Complicated Navigation Skills from Limited Experience Via Augmenting Offline Datasets

TendencyRL: Multi-stage Discriminative Hints for Efficient Goal-Oriented Reverse Curriculum Learning.