Abstract:The on-ramp merging driving scenario is challenging for achieving the highest-level autonomous driving. Current research using reinforcement learning methods to address the on-ramp merging problem of automated vehicles (AVs) is mainly designed for a single AV, treating other vehicles as part of the environment. This paper proposes a control framework for cooperative on-ramp merging of multiple AVs based on multi-agent deep reinforcement learning (MADRL). This framework facilitates AVs on the ramp and adjacent mainline to learn a coordinate control policy for their longitudinal and lateral motions based on the environment observations. Unlike the hierarchical architecture, this paper integrates decision and control into a unified optimal control problem to solve an on-ramp merging strategy through MADRL. Firstly, a partially observable Markov game (POMG) is formulated to characterize the on-ramp merging control problem, where the observation space of each AV (agent) is defined as its states and the relative state between it and other AVs, and the joint action spaces are the longitudinal acceleration and front wheel steering angle of AVs. Then, with safety and traffic efficiency as the objective, the reward function of each AV is designed. Furthermore, the joint action for multi-agent is obtained by solving the POMG problem utilizing the multi-agent deep deterministic policy gradient (MADDPG) method. Finally, a rule-based action guidance strategy is presented to supervise further the joint action for enhancing the safety of AVs. Numerical experiments are performed under different conditions to verify the effectiveness of the proposed merging control framework for a multi-agent system. The proposed scheme is also compared with the method for a single agent, taking the deep deterministic policy gradient (DDPG) method as a benchmark. The results demonstrate superior performance of the proposed method than the DDPG method in terms of safety and traffic efficiency.

Poster: Reliable On-Ramp Merging Via Multimodal Reinforcement Learning

Towards Robust On-Ramp Merging via Augmented Multimodal Reinforcement Learning

Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging

Autonomous On-ramp Merge Strategy Using Deep Reinforcement Learning in Uncertain Highway Environment

On-Ramp Merging for Highway Autonomous Driving: An Application of a New Safety Indicator in Deep Reinforcement Learning

A Deep Reinforcement Learning-Based Approach for Autonomous Driving in Highway On-Ramp Merge

Investigating Autonomous Vehicle Driving Strategies in Highway Ramp Merging Zones

Autonomous Highway Merging in Mixed Traffic Using Reinforcement Learning and Motion Predictive Safety Controller

Autonomous Ramp Merge Maneuver Based on Reinforcement Learning with Continuous Action Space

Reinforcement Learning with Latent State Inference for Autonomous On-ramp Merging under Observation Delay

A Deep Reinforcement Learning Approach for Automated On-Ramp Merging

Coordinated Longitudinal and Lateral Motions Control of Automated Vehicles Based on Multi-Agent Deep Reinforcement Learning for On-Ramp Merging

A Cooperative Control Algorithm for Real-time On-ramp Merging of Connected and Automated Vehicles

Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

Ramp Merging Strategy Based on Dual-layer Optimization Model in Non-connected Scenarios

On the Impact of Cooperative Autonomous Vehicles in Improving Freeway Merging: A Modified Intelligent Driver Model-Based Approach

Enhancing Cooperation of Vehicle Merging Control in Heavy Traffic Using Communication-Based Soft Actor-Critic Algorithm

Adaptive On-Ramp Merging Strategy under Imperfect Communication Performance

RACE-SM: Reinforcement Learning Based Autonomous Control for Social On-Ramp Merging

Collaborative Multi-Lane On-Ramp Merging Strategy for Connected and Automated Vehicles Using Dynamic Conflict Graph

Multi-Vehicle Cooperative Decision-Making in Merging Area Based on Deep Multi-Agent Reinforcement Learning