Abstract:Lane change decision-making is a complex task due to intricate vehicle-vehicle and vehicle-infrastructure interactions. Existing algorithms for lane-change control often depend on vehicles with a certain level of autonomy (e.g., autonomous or connected autonomous vehicles). To address the challenges posed by low penetration rates of autonomous vehicles and the high costs of precise data collection, this study proposes a dynamic lane change regulation design based on multi-agent reinforcement learning (MARL) to enhance freeway traffic efficiency. The proposed framework leverages multi-lane macroscopic traffic models that describe spatial-temporal dynamics of the density and speed for each lane. Lateral traffic flow between adjacent lanes, resulting from aggregated lane-changing behaviors, is modeled as source terms exchanged between the partial differential equations (PDEs). We propose a lane change regulation strategy using MARL, where one agent is placed at each discretized lane grid. The state of each agent is represented by aggregated vehicle attributes within its grid, generated from the SUMO microscopic simulation environment. The agent's actions are lane-change regulations for vehicles in its grid. Specifically, lane-change regulation signals are computed at a centralized traffic management center and then broadcast to connected vehicles in the corresponding lane grids. Compared to vehicle-level maneuver control, this approach achieves a higher regulation rate by leveraging vehicle connectivity while introducing no critical safety concerns, and accommodating varying levels of connectivity and autonomy within the traffic system. The proposed model is simulated and evaluated in varied traffic scenarios and demand conditions. Experimental results demonstrate that the method improves overall traffic efficiency with minimal additional energy consumption while maintaining driving safety.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to improve the traffic efficiency on expressways, especially in terms of lane - change decisions. Specifically, the paper proposes solutions to the following key challenges: 1. **Low - penetration - rate Connected and Autonomous Vehicles (CAVs)**: Currently, the penetration rate of autonomous vehicles is low, which limits the effectiveness of existing lane - change control algorithms. Therefore, a method that can work effectively in a mixed - traffic environment is required, which can improve the overall traffic efficiency even when the number of CAVs is small. 2. **High - cost data collection**: Accurate data collection is crucial for traditional lane - change control algorithms, but it is often costly or even infeasible in large - scale environments. Therefore, a method that reduces the dependence on accurate data is needed. To solve these problems, the paper proposes a dynamic lane - change regulation design based on multi - agent reinforcement learning (MARL). This method uses a macroscopic traffic model to describe the spatio - temporal density and speed dynamics of each lane, and models the lateral traffic flow between adjacent lanes as a source - term exchange between partial differential equations (PDEs). Specifically, the main contributions of this study include: 1. **Proposing a new lane - change regulation method based on lane grids**: Through an infrastructure - level framework, regulate lane - change behaviors at the macroscopic level to improve the overall efficiency of mixed expressway traffic. 2. **Simplifying communication requirements and reducing the dependence on accurate vehicle positioning**: Compared with traditional MARL - based vehicle control methods, this strategy can be more efficiently applied to large - scale traffic systems. 3. **Establishing a hierarchical reinforcement learning framework**: Combine a microscopic simulator to generate microscopic vehicle trajectories and multi - lane macroscopic traffic PDEs to calculate aggregated lane - change decisions, thereby improving expressway traffic performance. ### Formula Summary - **Lighthill - Whitham - Richards (LWR) Model**: \[ \frac{\partial \rho}{\partial t}+\frac{\partial f}{\partial x} = 0 \] where $\rho$ represents vehicle density, $f$ represents traffic flow, $t$ is time, and $x$ is spatial position. - **Lane - density evolution equation with source terms**: \[ \frac{\partial \rho_\alpha}{\partial t}+\frac{\partial f_\alpha}{\partial x}=G_{\text{ain}}^\alpha - L_{\text{oss}}^\alpha \] - **Lane - density evolution equation with regulation signals**: \[ \frac{\partial \rho_\alpha}{\partial t}+\frac{\partial f_\alpha}{\partial x}=(\rho_{\alpha - 1}\frac{1}{T_{\text{left}}}a_{\text{left}}^{\alpha - 1}+\rho_{\alpha + 1}\frac{1}{T_{\text{right}}}a_{\text{right}}^{\alpha + 1})-(\rho_\alpha\frac{1}{T_{\text{left}}}a_{\text{left}}^\alpha+\rho_\alpha\frac{1}{T_{\text{right}}}a_{\text{right}}^\alpha) \] - **Reward function**: \[ r_1=\frac{1}{|B(\alpha_i)|}\sum_{\tilde{\alpha}_k\in B(\alpha_i)}\frac{v_{\tilde{\alpha}_k}}{v_{\text{max}}} \] \[ r_2=\frac{1}{|B|}

Reinforcement Learning for Freeway Lane-Change Regulation via Connected Vehicles

A Reinforcement Learning Approach to Smart Lane Changes of Self-driving Cars

Developing Smart Lane-changing Strategies for CAVs on Freeways based on MOBIL and Reinforcement Learning

A Q-learning Foresighted Approach to Ego-efficient Lane Changes of Connected and Automated Vehicles on Freeways.

Modeling Heterogeneous Traffic at Ramp with Autonomous Vehicles Based on Cellular Automata and Multi-Agent System

LCS-TF: Multi-Agent Deep Reinforcement Learning-Based Intelligent Lane-Change System for Improving Traffic Flow

Decentralized Cooperative Lane Changing at Freeway Weaving Areas Using Multi-Agent Deep Reinforcement Learning

Multi-Agent Deep Reinforcement Learning for Multi-Lane Freeways Differential Variable Speed Limit Control in Mixed Traffic Environment

Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Multi-Task Lane-Free Driving Strategy for Connected and Automated Vehicles: A Multi-Agent Deep Reinforcement Learning Approach

A multi-agent reinforcement learning-based longitudinal and lateral control of CAVs to improve traffic efficiency in a mandatory lane change scenario

An Evolutionary Learning Framework of Lane-Changing Control for Autonomous Vehicles at Freeway Off-Ramps

Reinforcement Learning from Human Feedback for Lane Changing of Autonomous Vehicles in Mixed Traffic

An Improved Hierarchical Deep Reinforcement Learning Algorithm for Multi-Intelligent Vehicle Lane Change

Chance-Aware Lane Change with High-Level Model Predictive Control Through Curriculum Reinforcement Learning

A New Solution for Freeway Congestion: Cooperative Speed Limit Control Using Distributed Reinforcement Learning

Leveraging the Capabilities of Connected and Autonomous Vehicles and Multi-Agent Reinforcement Learning to Mitigate Highway Bottleneck Congestion

Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning

A Novel Dynamic Lane-Changing Trajectory Planning Model for Automated Vehicles Based on Reinforcement Learning

Coordinated Variable Speed Limit Control for Consecutive Bottlenecks on Freeways Using Multiagent Reinforcement Learning

Discretionary Lane-Change Decision and Control via Parameterized Soft Actor–Critic for Hybrid Action Space