Abstract:With the advancement of Intelligent Transportation Systems and Vehicle-to-Everything communication technologies, the future traffic scenario is anticipated to be a mixed environment consisting of heterogeneous Connected and Autonomous Vehicles (CAVs) and Human-Driven Vehicles. Though reinforcement learning (RL) has shown tremendous potential in solving decision-making problems in autonomous driving, existing research often overlooks the heterogeneity of CAVs. Moreover, current RL methods lack robust safety guarantees, especially in mixed traffic scenarios where HDVs are often non-cooperative and uncontrollable. To address challenges of the cooperative driving among heterogeneous CAVs in dynamic and mixed traffic, we have developed a novel heterogeneous cluster safe algorithm and hybrid-critic training framework. Specifically, to overcome the limitations of existing methods primarily designed for homogeneous agents, we introduce a cluster policy-sharing mechanism. Additionally, we propose a safe policy update method with a high sample utilization rate, which combined with the cluster policy-sharing mechanism leads to the HCP4O algorithm. Furthermore, we have designed an innovative hybrid-critic training framework to enhance the safety driving performance of heterogeneous CAVs under various conditions. Finally, we train our model in hundreds of random and dynamic scenarios to validate the generalization ability of agents. The experimental results show that, compared to current advanced Multi-Agent RL algorithms and rule-based methods, our approach demonstrates significant advantages in safety and efficiency, especially in scenarios with a large number of agents.

Hierarchical Clustering-Based State Grouping Reinforcement Learning for Switching Decision of Autonomous Vehicles

Learning Hierarchical Behavior and Motion Planning for Autonomous Driving.

Safe-State Enhancement Method for Autonomous Driving Via Direct Hierarchical Reinforcement Learning.

Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data

Hierarchical Reinforcement Learning-Based Policy Switching Towards Multi-Scenarios Autonomous Driving

Boosting Reinforcement Learning via Hierarchical Game Playing With State Relay

Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data

Multi-Task Long-Range Urban Driving Based on Hierarchical Planning and Reinforcement Learning

Generalized Single-Vehicle-Based Graph Reinforcement Learning for Decision-Making in Autonomous Driving

Towards Cooperative Driving among Heterogeneous CAVs: A Safe Multi-Agent Reinforcement Learning Approach

Spatially and Seamlessly Hierarchical Reinforcement Learning for State Space and Policy space in Autonomous Driving

An Improved Reinforcement Learning Method Based on Unsupervised Learning

Multi-Agent Reinforcement Learning-Based Decision Making for Twin-Vehicles Cooperative Driving in Stochastic Dynamic Highway Environments

Deep Hierarchical Reinforcement Learning for Autonomous Driving with Distinct Behaviors.

Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation

Autonomous Lane Change Decision Making Using Different Deep Reinforcement Learning Methods

A Safe Hierarchical Planning Framework for Complex Driving Scenarios based on Reinforcement Learning

Autonomous Surface Vessel Obstacle Avoidance Based on Hierarchical Reinforcement Learning

Decision-making in Autonomous Driving by Reinforcement Learning Combined with Planning & Control

Behavior Planning at Urban Intersections through Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning for Autonomous Decision Making and Motion Planning of Intelligent Vehicles