Optimization Control of Adaptive Traffic Signal with Deep Reinforcement Learning

Kerang Cao,Liwei Wang,Shuo Zhang,Lini Duan,Guimin Jiang,Stefano Sfarra,Hai Zhang,Hoekyung Jung

DOI: https://doi.org/10.3390/electronics13010198

IF: 2.9

2024-01-02

Electronics

Abstract:The optimization and control of traffic signals is very important for logistics transportation. It not only improves the operational efficiency and safety of road traffic, but also conforms to the direction of the intelligent, green, and sustainable development of modern cities. In order to improve the optimization effect of traffic signal control, this paper proposes a traffic signal optimization method based on deep reinforcement learning and Simulation of Urban Mobility (SUMO) software for urban traffic scenarios. The intersection training scenario was established using SUMO micro traffic simulation software, and the maximum vehicle queue length and vehicle queue time were selected as performance evaluation indicators. In order to be more relevant to the real environment, the experiment uses Weibull distribution to simulate vehicle generation. Since deep reinforcement learning takes into account both perceptual and decision-making capabilities, this study proposes a traffic signal optimization control model based on the deep reinforcement learning Deep Q Network (DQN) algorithm by considering the realism and complexity of traffic intersections, and first uses the DQN algorithm to train the model in a training scenario. After that, the G-DQN (Grouping-DQN) algorithm is proposed to address the problems that the definition of states in existing studies cannot accurately represent the traffic states and the slow convergence of neural networks. Finally, the performance of the G-DQN algorithm model was compared with the original DQN algorithm model and Advantage Actor-Critic (A2C) algorithm model. The experimental results show that the improved algorithm increased the main indicators in all aspects.

engineering, electrical & electronic,computer science, information systems,physics, applied

What problem does this paper attempt to address?

This paper mainly discusses the problem of urban traffic signal optimization control, using Deep Reinforcement Learning (DRL) methods to improve adaptive traffic signal control. In the study, the authors built a training model for intersection scenes in urban traffic based on the Simulation of Urban Mobility (SUMO) software, using maximum vehicle queue length and vehicle queueing time as performance evaluation indicators. To make the experiments more realistic, the Weibull distribution was used to simulate vehicle generation.

The paper proposes a traffic signal optimization control model based on deep reinforcement learning, using the Deep Q Network (DQN) algorithm for training. To address the issues of inaccurate reflection of traffic states and slow convergence speed of neural networks in existing research, the Grouping-DQN (G-DQN) algorithm is proposed. G-DQN redefines the states and rewards, and improves the CNN network structure to more accurately reflect the state of traffic intersections, thereby improving the model performance.

By comparing with the original DQN algorithm model and the Advantage Actor-Critic (A2C) algorithm model, the experimental results show that the improved G-DQN algorithm has improved in various key indicators. The contributions of the paper are:

1. Speeding up model training and effectively reducing the number of vehicle queues.
2. Proposing the G-DQN algorithm, redefining the states and rewards in reinforcement learning, using a dual-channel CNN structure to improve model performance, surpassing DQN and A2C algorithms.
3. Using SUMO simulation software for modeling and simulation, and using the Weibull distribution to simulate traffic flow in real environments.

The last part of the paper introduces related work, including the SUMO traffic simulation software, basic concepts of deep reinforcement learning, and experimental design and data generation methods.