Reinforcement Learning with Model Predictive Control for Highway Ramp Metering

Filippo Airaldi,Bart De Schutter,Azita Dabiri

2024-10-24

Abstract:In the backdrop of an increasingly pressing need for effective urban and highway transportation systems, this work explores the synergy between model-based and learning-based strategies to enhance traffic flow management by use of an innovative approach to the problem of ramp metering control that embeds Reinforcement Learning (RL) techniques within the Model Predictive Control (MPC) framework. The control problem is formulated as an RL task by crafting a suitable stage cost function that is representative of the traffic conditions, variability in the control action, and violations of the constraint on the maximum number of vehicles in queue. An MPC-based RL approach, which leverages the MPC optimal problem as a function approximation for the RL algorithm, is proposed to learn to efficiently control an on-ramp and satisfy its constraints despite uncertainties in the system model and variable demands. Simulations are performed on a benchmark small-scale highway network to compare the proposed methodology against other state-of-the-art control approaches. Results show that, starting from an MPC controller that has an imprecise model and is poorly tuned, the proposed methodology is able to effectively learn to improve the control policy such that congestion in the network is reduced and constraints are satisfied, yielding an improved performance that is superior to the other controllers.

Systems and Control,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively balance the two often - conflicting goals of traffic congestion on the main road and the ramp queue length in highway on - ramp control. Specifically, the paper explores the combination of model - based methods (such as model predictive control, MPC) and learning - based methods (such as reinforcement learning, RL) to improve the efficiency of traffic flow management. By embedding RL techniques within the MPC framework, the paper proposes a new method to solve the on - ramp control problem, aiming to overcome the limitations of traditional MPC methods in terms of sensitivity to model inaccuracies and the need for precise parameter identification, while taking advantage of the learning ability of RL algorithms to adapt to system uncertainties and changing requirements. The core problem of the paper is to design a method that can effectively control the traffic flow entering the highway network. This method can optimize the control strategy through learning in the case of system model uncertainties, reduce congestion in the network, and meet constraints, such as the maximum number of queuing vehicles limit. In this way, the paper proposes an innovative solution to improve the effectiveness of traffic flow management and on - ramp control.

Reinforcement Learning with Model Predictive Control for Highway Ramp Metering

Reinforcement Learning with Model Predictive Control for Highway Ramp Metering

An Adaptive Control Scheme for Local and Coordinated Ramp Metering

A Physics-Informed Reinforcement Learning-Based Strategy for Local and Coordinated Ramp Metering

A Machine Learning Method for Dynamic Traffic Control and Guidance on Freeway Networks

A Cyber-Physical System for Freeway Ramp Meter Signal Control Using Deep Reinforcement Learning in a Connected Environment

A Deep Reinforcement Learning Approach for Ramp Metering Based on Traffic Video Data

A Novel Framework Combining MPC and Deep Reinforcement Learning With Application to Freeway Traffic Control

An Extended Model Predictive Control Approach To Coordinated Ramp Metering

Coordinated Ramp Metering with Equity Consideration Using Reinforcement Learning

Demonstration-guided Deep Reinforcement Learning for Coordinated Ramp Metering and Perimeter Control in Large Scale Networks

Freeway network traffic management based on distributed reinforcement learning

Enhancing reinforcement learning‐based ramp metering performance at freeway uncertain bottlenecks using curriculum learning

Output-feedback adaptive model predictive control for ramp metering: a set-membership approach

A Complementary Modularized Ramp Metering Approach Based on Iterative Learning Control and ALINEA

Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging

Prediction Based Active Ramp Metering Control Strategy with Mobility and Safety Assessment

Coordinated Control of Urban Expressway Networks Via MPC

A Freeway Traffic Flow Control Model Based on Distributed Reinforcement Learning

A Highway Entrance Ramp Control Method Based on Deep Reinforcement Learning