Integrated Freeway Traffic Control Using Q-Learning with Adjacent Arterial Traffic Considerations

Tianchen Yuan,Petros A. Ioannou

2024-05-20

Abstract:Numerous studies have shown the effectiveness of intelligent transportation system techniques such as variable speed limit (VSL), lane change (LC) control, and ramp metering (RM) in freeway traffic flow control. The integration of these techniques has the potential to further enhance the traffic operation efficiency of both freeway and adjacent arterial networks. In this regard, we propose a freeway traffic control (FTC) strategy that coordinates VSL, LC, RM actions using a Q-learning (QL) framework which takes into account arterial traffic characteristics. The signal timing and demands of adjacent arterial intersections are incorporated as state variables of the FTC agent. The FTC agent is initially trained offline using a single-section road network, and subsequently deployed online in a connected freeway and arterial simulation network for continuous learning. The arterial network is assumed to be regulated by a traffic-responsive signal control strategy based on a cycle length model. Microscopic simulations demonstrate that the fully-trained FTC agent provides significant reductions in freeway travel time and the number of stops in scenarios with traffic congestion. It clearly outperforms an uncoordinated FTC and a decentralized feedback control strategy. Even though the FTC agent does not control the arterial traffic signals, it leads to shorter average queue lengths at arterial intersections by taking into account the arterial traffic conditions in controlling freeway traffic. These results motivate a future research where the QL framework will also include the control of arterial traffic signals.

Systems and Control

What problem does this paper attempt to address?

This paper proposes a highway traffic control strategy that integrates Variable Speed Limit (VSL), Lane Change (LC) control, and Ramp Metering (RM) while considering the traffic conditions of adjacent arterials using the Q-learning (QL) framework. In this study, the authors treat signal timing and demand at the intersections of adjacent arterials as the state variables of the Freeway Traffic Control (FTC) agent, and pre-train the agent in a single-road segment road network offline, and then continuously learn in the connected highway and arterial simulation network online. The problem addressed in the paper is how to effectively integrate different traffic control techniques to improve the operational efficiency of highways and adjacent arterial networks. Traditional control frameworks face scalability issues or lack coordination, while reinforcement learning-based frameworks can enhance coordination and adapt to large-scale road networks. The paper designs a Reinforcement Learning (RL) algorithm by combining VSL, LC, and RM, and considering arterial traffic conditions, aiming to reduce travel time and the number of stops in congested traffic conditions, while also reducing the average queue length at arterial intersections. The findings of the study indicate significant improvements compared to uncoordinated control strategies in congested traffic scenarios. Although this strategy does not directly control arterial traffic signals, it indirectly reduces the average queue length at arterial intersections by considering arterial traffic conditions to control highway traffic. Future research plans to expand the Q-learning framework to include arterial traffic signal control.

Integrated Freeway Traffic Control Using Q-Learning with Adjacent Arterial Traffic Considerations

Q-learning Based Traffic Signal Control Method for an Isolated Intersection

Traffic Signal Control and Speed Offset Coordination Using Q-Learning for Arterial Road Networks

A Q-learning Foresighted Approach to Ego-efficient Lane Changes of Connected and Automated Vehicles on Freeways.

Integrated Control of Internal Boundary and Ramp Inflows for Lane-free Traffic of Automated Vehicles on Freeways.

Integrated Variable Speed Limits and Lane-Changing Control for Freeway Lane-Drop Bottlenecks

Multi-Agent Deep Reinforcement Learning for Multi-Lane Freeways Differential Variable Speed Limit Control in Mixed Traffic Environment

Traffic Scheduling and Control in Fully Connected and Automated Networks

Coordinated Variable Speed Limit Control for Consecutive Bottlenecks on Freeways Using Multiagent Reinforcement Learning

Reinforcement Learning for Freeway Lane-Change Regulation via Connected Vehicles

Integrated Traffic Control for Freeway Recurrent Bottleneck Based on Deep Reinforcement Learning

A New Solution for Freeway Congestion: Cooperative Speed Limit Control Using Distributed Reinforcement Learning

Integrated control of highway traffic flow

A scalable approach to optimize traffic signal control with federated reinforcement learning

OAS Deep Q-Learning-Based Fast and Smooth Control Method for Traffic Signal Transition in Urban Arterial Tidal Lanes

A Novel Framework Combining MPC and Deep Reinforcement Learning With Application to Freeway Traffic Control

MARL-FWC: Optimal Coordination of Freeway Traffic Control Measures

An Information Fusion Approach to Intelligent Traffic Signal Control Using the Joint Methods of Multiagent Reinforcement Learning and Artificial Intelligence of Things

A Cyber-Physical System for Freeway Ramp Meter Signal Control Using Deep Reinforcement Learning in a Connected Environment

Coordinated Ramp Metering and Intersection Signal Control

Learning in practice: reinforcement learning-based traffic signal control augmented with actuated control