Abstract:Hierarchical reinforcement learning (HRL) incorporates temporal abstraction into reinforcement learning (RL) by explicitly taking advantage of hierarchical structure. Modern HRL typically designs a hierarchical agent composed of a high-level policy and low-level policies. The high-level policy selects which low-level policy to activate at a lower frequency and the activated low-level policy selects an action at each time step. Recent HRL algorithms have achieved performance gains over standard RL algorithms in synthetic navigation tasks. However, we cannot apply these HRL algorithms to real-world navigation tasks. One of the main challenges is that real-world navigation tasks require an agent to perform safe and interactive behaviors in dynamic environments. In this paper, we propose imagination-augmented HRL (IAHRL) that efficiently integrates imagination into HRL to enable an agent to learn safe and interactive behaviors in real-world navigation tasks. Imagination is to predict the consequences of actions without interactions with actual environments. The key idea behind IAHRL is that the low-level policies imagine safe and structured behaviors, and then the high-level policy infers interactions with surrounding objects by interpreting the imagined behaviors. We also introduce a new attention mechanism that allows our high-level policy to be permutation-invariant to the order of surrounding objects and to prioritize our agent over them. To evaluate IAHRL, we introduce five complex urban driving tasks, which are among the most challenging real-world navigation tasks. The experimental results indicate that IAHRL enables an agent to perform safe and interactive behaviors, achieving higher success rates and lower average episode steps than baselines.

Cola-HRL: Continuous-Lattice Hierarchical Reinforcement Learning for Autonomous Driving

Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data

Safe-State Enhancement Method for Autonomous Driving Via Direct Hierarchical Reinforcement Learning.

Learning Hierarchical Behavior and Motion Planning for Autonomous Driving.

Large Language Model guided Deep Reinforcement Learning for Decision Making in Autonomous Driving

HGRL: Human-Driving-Data Guided Reinforcement Learning for Autonomous Driving

Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data

Imagination-Augmented Hierarchical Reinforcement Learning for Safe and Interactive Autonomous Driving in Urban Environments

Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving

Multi-Task Long-Range Urban Driving Based on Hierarchical Planning and Reinforcement Learning

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

Action and Trajectory Planning for Urban Autonomous Driving with Hierarchical Reinforcement Learning

An Integrated Framework of Lateral and Longitudinal Behavior Decision-Making for Autonomous Driving Using Reinforcement Learning

A Hierarchical Framework for Multi-Lane Autonomous Driving Based on Reinforcement Learning

Task-Driven Autonomous Driving: Balanced Strategies Integrating Curriculum Reinforcement Learning and Residual Policy

Continuous Reinforcement Learning From Human Demonstrations With Integrated Experience Replay For Autonomous Driving

Self-Adaptive Driving in Nonstationary Environments through Conjectural Online Lookahead Adaptation

Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learning

Automated Driving Maneuvers under Interactive Environment based on Deep Reinforcement Learning

VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving

Applications of Distributional Soft Actor-Critic in Real-world Autonomous Driving