On-Ramp Merging for Highway Autonomous Driving: An Application of a New Safety Indicator in Deep Reinforcement Learning

Guofa Li,Weiyan Zhou,Siyan Lin,Shen Li,Xingda Qu

DOI: https://doi.org/10.1007/s42154-023-00235-2

2023-08-01

Automotive Innovation

Abstract:This paper proposes an improved decision-making method based on deep reinforcement learning to address on-ramp merging challenges in highway autonomous driving. A novel safety indicator, time difference to merging (TDTM), is introduced, which is used in conjunction with the classic time to collision (TTC) indicator to evaluate driving safety and assist the merging vehicle in finding a suitable gap in traffic, thereby enhancing driving safety. The training of an autonomous driving agent is performed using the Deep Deterministic Policy Gradient (DDPG) algorithm. An action-masking mechanism is deployed to prevent unsafe actions during the policy exploration phase. The proposed DDPG + TDTM + TTC solution is tested in on-ramp merging scenarios with different driving speeds in SUMO and achieves a success rate of 99.96% without significantly impacting traffic efficiency on the main road. The results demonstrate that DDPG + TDTM + TTC achieved a higher on-ramp merging success rate of 99.96% compared to DDPG + TTC and DDPG.

engineering, mechanical, electrical & electronic,transportation science & technology

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the decision - making challenge of ramp merging in highway autonomous driving. Specifically, ramp merging is a high - risk road traffic scenario, with nearly 300,000 accidents and 50,000 deaths in the United States each year. Since ramp merging needs to be completed at a safe speed within a limited distance, it has become one of the most challenging decision - making scenarios for autonomous vehicles. Therefore, solving the decision - making challenges in ramp merging is crucial for achieving safe autonomous driving. To address this challenge, the paper proposes an improved decision - making method based on deep reinforcement learning (DRL), introducing a new safety metric - Time Difference to the Merge Point (TDTM), which is used in combination with the classic Time - to - Collision (TTC) metric to evaluate driving safety and help merging vehicles find suitable traffic gaps, thereby improving driving safety. By using the Deep Deterministic Policy Gradient (DDPG) algorithm to train the autonomous driving agent, and deploying an action - masking mechanism to prevent unsafe actions during the policy exploration phase. The experimental results show that the proposed DDPG + TDTM + TTC solution has achieved a 99.96% success rate in ramp - merging scenarios at different driving speeds, and has little impact on the traffic efficiency of the main road.

On-Ramp Merging for Highway Autonomous Driving: An Application of a New Safety Indicator in Deep Reinforcement Learning

Autonomous On-ramp Merge Strategy Using Deep Reinforcement Learning in Uncertain Highway Environment

Autonomous Highway Merging in Mixed Traffic Using Reinforcement Learning and Motion Predictive Safety Controller

Learning an Efficient and Safe Policy for Highway Driving Using Supervised Learning and Reinforcement Learning.

A Deep Reinforcement Learning-Based Approach for Autonomous Driving in Highway On-Ramp Merge

Investigating Autonomous Vehicle Driving Strategies in Highway Ramp Merging Zones

A Roadside Decision-Making Methodology Based on Deep Reinforcement Learning to Simultaneously Improve the Safety and Efficiency of Merging Zone.

A Deep Reinforcement Learning Approach for Automated On-Ramp Merging

Leveraging on Deep Reinforcement Learning for Autonomous Safe Decision-Making in Highway On-ramp Merging (Student Abstract)

Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging

Autonomous Ramp Merge Maneuver Based on Reinforcement Learning with Continuous Action Space

Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge

Poster: Reliable On-Ramp Merging Via Multimodal Reinforcement Learning

Novel Decision-Making Strategy for Connected and Autonomous Vehicles in Highway On-Ramp Merging

Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

High-Speed Ramp Merging Behavior Decision for Autonomous Vehicles Based on Multi-Agent Reinforcement Learning

Coordinated Longitudinal and Lateral Motions Control of Automated Vehicles Based on Multi-Agent Deep Reinforcement Learning for On-Ramp Merging

Reinforcement Learning with Latent State Inference for Autonomous On-ramp Merging under Observation Delay

Decision-Making under On-Ramp merge Scenarios by Distributional Soft Actor-Critic Algorithm

Graph-based multi agent reinforcement learning for on-ramp merging in mixed traffic

Distributional Soft Actor-Critic for Decision-Making in On-Ramp Merge Scenarios