Abstract:The on-ramp merging driving scenario is challenging for achieving the highest-level autonomous driving. Current research using reinforcement learning methods to address the on-ramp merging problem of automated vehicles (AVs) is mainly designed for a single AV, treating other vehicles as part of the environment. This paper proposes a control framework for cooperative on-ramp merging of multiple AVs based on multi-agent deep reinforcement learning (MADRL). This framework facilitates AVs on the ramp and adjacent mainline to learn a coordinate control policy for their longitudinal and lateral motions based on the environment observations. Unlike the hierarchical architecture, this paper integrates decision and control into a unified optimal control problem to solve an on-ramp merging strategy through MADRL. Firstly, a partially observable Markov game (POMG) is formulated to characterize the on-ramp merging control problem, where the observation space of each AV (agent) is defined as its states and the relative state between it and other AVs, and the joint action spaces are the longitudinal acceleration and front wheel steering angle of AVs. Then, with safety and traffic efficiency as the objective, the reward function of each AV is designed. Furthermore, the joint action for multi-agent is obtained by solving the POMG problem utilizing the multi-agent deep deterministic policy gradient (MADDPG) method. Finally, a rule-based action guidance strategy is presented to supervise further the joint action for enhancing the safety of AVs. Numerical experiments are performed under different conditions to verify the effectiveness of the proposed merging control framework for a multi-agent system. The proposed scheme is also compared with the method for a single agent, taking the deep deterministic policy gradient (DDPG) method as a benchmark. The results demonstrate superior performance of the proposed method than the DDPG method in terms of safety and traffic efficiency.

A Cooperative Control Methodology Considering Dynamic Interaction for Multiple Connected and Automated Vehicles in the Merging Zone

Cooperative On-Ramp Merging Control of Connected and Automated Vehicles: Distributed Multi-Agent Deep Reinforcement Learning Approach

Coordinated Longitudinal and Lateral Motions Control of Automated Vehicles Based on Multi-Agent Deep Reinforcement Learning for On-Ramp Merging

A Hierarchical Model-Based Optimization Control Approach for Cooperative Merging by Connected Automated Vehicles

Enhancing Cooperation of Vehicle Merging Control in Heavy Traffic Using Communication-Based Soft Actor-Critic Algorithm

Connected Automated Vehicle Cooperative Control with a Deep Reinforcement Learning Approach in a Mixed Traffic Environment

A Multi-Agent Deep Reinforcement Learning Coordination Framework for Connected and Automated Vehicles at Merging Roadways

Safety-Critical and Flexible Cooperative On-Ramp Merging Control of Connected and Automated Vehicles in Mixed Traffic

Empirical Study of a Cooperative Longitudinal Control for Merging Maneuvers Considering Courtesy and Mixed Autonomy

Multi-Vehicle Cooperative Decision-Making in Merging Area Based on Deep Multi-Agent Reinforcement Learning

Cooperative Highway Work Zone Merge Control Based on Reinforcement Learning in a Connected and Automated Environment

Research on Cooperative Control for On-Ramp Merging with Multiple Lanes in Connected and Automated Environments

Game-Based Flexible Merging Decision Method for Mixed Traffic of Connected Autonomous Vehicles and Manual Driving Vehicles on Urban Freeways

Connected and Automated Vehicle Distributed Control for On-ramp Merging Scenario: A Virtual Rotation Approach

A Homogeneous Multi-Vehicle Cooperative Group Decision-Making Method in Complicated Mixed Traffic Scenarios

Cooperative Group Control Strategy in the on-Ramp Area for Connected and Automated Vehicles under Mixed Traffic Environment

A cooperative on-ramp merging strategy based on the hybrid of centralized and distributed interaction for heterogeneous vehicles

A Cooperative Control Algorithm for Real-time On-ramp Merging of Connected and Automated Vehicles

Cooperative Lane‐Changing Control for CAVs at Freeway On‐Ramps considering Vehicle Dynamics

Embedding Robust Constraint-Following Control in Cooperative On-Ramp Merging

A Grouping-Based Cooperative Driving Strategy for CAVs Merging Problems.