Abstract:This study focuses on a crucial task in the field of autonomous driving, autonomous lane change. Autonomous lane change plays a pivotal role in improving traffic flow, alleviating driver burden, and reducing the risk of traffic accidents. However, due to the complexity and uncertainty of lane-change scenarios, the functionality of autonomous lane change still faces challenges. In this research, we conducted autonomous lane-change simulations using both deep reinforcement learning (DRL) and model predictive control (MPC). Specifically, we used the parameterized soft actor--critic (PASAC) algorithm to train a DRL-based lane-change strategy to output both discrete lane-change decisions and continuous longitudinal vehicle acceleration. We also used MPC for lane selection based on the smallest predictive car-following costs for the different lanes. For the first time, we compared the performance of DRL and MPC in the context of lane-change decisions. The simulation results indicated that, under the same reward/cost function and traffic flow, both MPC and PASAC achieved a collision rate of 0%. PASAC demonstrated a comparable performance to MPC in terms of average rewards/costs and vehicle speeds.

What problem does this paper attempt to address?

The paper attempts to address the problem of achieving autonomous lane-changing decision-making and control in self-driving vehicles. Specifically, it focuses on how to enable self-driving vehicles to safely and efficiently perform lane-changing maneuvers in complex traffic environments using Deep Reinforcement Learning (DRL) and Model Predictive Control (MPC) techniques. ### Background and Problem 1. **Importance and Challenges of Autonomous Driving**: - Autonomous driving technology can reduce driver burden, improve traffic flow, and significantly decrease traffic accidents caused by human error. - According to the World Health Organization, approximately 1.3 million people die in road traffic accidents globally each year, with 94% of these accidents caused by driver error. - In lane-changing scenarios, the behavior of surrounding vehicles is difficult to predict, making autonomous lane-changing a critical task for self-driving vehicles. 2. **Limitations of Existing Research**: - Currently, most automated lane-changing functions are still at the L2 level of automation. Despite some progress, they have not been widely applied in real vehicles. - Most literature uses discrete reinforcement learning to achieve non-mandatory automated lane-changing control, but few studies have applied reinforcement learning with hybrid action spaces to this field. ### Research Methodology 1. **Technical Approach**: - The paper uses the Parameterized Soft Actor-Critic (PASAC) algorithm, a deep reinforcement learning algorithm suitable for hybrid action spaces. - Additionally, the paper employs the Model Predictive Control (MPC) method for comparative study. 2. **Experimental Design**: - Traffic simulations were conducted on the SUMO platform, simulating various traffic scenarios. - By comparing the performance of PASAC and MPC in terms of collision rate, average speed, value function, and acceleration changes, the advantages of the PASAC algorithm were validated. ### Main Contributions 1. **Introduction of PASAC Algorithm**: - This study is the first to apply the PASAC algorithm to autonomous lane-changing decision-making and control. - Through quantitative and comprehensive comparisons, the superiority of the PASAC algorithm in lane-changing decision-making and control was validated. 2. **Performance Evaluation**: - Experimental results show that PASAC outperforms MPC in terms of average speed and cost, and executes more lane changes. - A detailed analysis of PASAC's performance provides new insights and empirical support. ### Conclusion By introducing the PASAC algorithm, this paper successfully addresses the problem of autonomous lane-changing in complex traffic environments for self-driving vehicles. The study not only provides new ideas for the application of reinforcement learning with hybrid action spaces in practical control problems but also offers empirical support for the comparison between reinforcement learning and traditional control methods.

Discretionary Lane-Change Decision and Control via Parameterized Soft Actor-Critic for Hybrid Action Space

Discretionary Lane-Change Decision and Control via Parameterized Soft Actor–Critic for Hybrid Action Space

Safe Hybrid-Action Reinforcement Learning-Based Decision and Control for Discretionary Lane Change

A Reinforcement Learning Approach to Smart Lane Changes of Self-driving Cars

Highway Discretionary Lane-change Decision and Control Using Model Predictive Control

Highway Decision-Making and Motion Planning for Autonomous Driving via Soft Actor-Critic

Chance-Aware Lane Change with High-Level Model Predictive Control Through Curriculum Reinforcement Learning

Integrated eco-driving automation of intelligent vehicles in multi-lane scenario via model-accelerated reinforcement learning

Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning

Mandatory Lane-Changing Decision-Making in Dense Traffic for Autonomous Vehicles Based on Deep Reinforcement Learning

A Deep Reinforcement Learning-Based Approach for Autonomous Lane-Changing Velocity Control in Mixed Flow of Vehicle Group Level

Combining Decision Making and Trajectory Planning for Lane Changing Using Deep Reinforcement Learning

Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithm

A Safe and Efficient Lane Change Decision-Making Strategy of Autonomous Driving Based on Deep Reinforcement Learning

Autonomous lane-change controller

Deep Reinforcement Learning Lane-Changing Decision Algorithm for Intelligent Vehicles Combining LSTM Trajectory Prediction

Decision Making of Autonomous Vehicles in Lane Change Scenarios: Deep Reinforcement Learning Approaches with Risk Awareness

End-to-End Automated Lane-Change Maneuvering Considering Driving Style Using a Deep Deterministic Policy Gradient Algorithm

A Model Predictive Controller for Automated Lane Change Maneuver with Multi-Constraints

Reinforcement Learning from Human Feedback for Lane Changing of Autonomous Vehicles in Mixed Traffic

Driving Decision and Control for Autonomous Lane Change based on Deep Reinforcement Learning