[The vaccines of the future].

M. Girard

1987-07-01

Abstract:

What problem does this paper attempt to address?

Graph cooperation deep reinforcement learning for ecological urban traffic signal control

Liping Yan,Lulong Zhu,Kai Song,Zhaohui Yuan,Yunjuan Yan,Yue Tang,Chan Peng

DOI: https://doi.org/10.1007/s10489-022-03208-w

IF: 5.3

2022-07-07

Applied Intelligence

Abstract:Cooperation between intersections in large-scale road networks is critical in traffic congestion. Currently, most traffic signals cooperate via pre-defined timing phases, which is extremely inefficient in real-time traffic scenarios. Most existing studies on multi-agent reinforcement learning (MARL) traffic signal control have focused on designing efficient communication methods, but have ignored the importance of how agents interact in cooperative communication. To achieve more efficient cooperation among traffic signals and alleviate urban traffic congestion, this study constructs a Graph Cooperation Q-learning Network Traffic Signal Control (GCQN-TSC) model, which is a graph cooperation network with an embedded self-attention mechanism that enables agents to adjust their attention in real time according to the dynamic traffic flow information, perceive the traffic environment quickly and effectively in a larger range, and help agents achieve more effective collaboration. Moreover, the Deep Graph Q-learning (DGQ) algorithm is proposed in this model to optimize the traffic signal control strategy according to the spatio-temporal characteristics of different traffic scenes and provide the optimal signal phase for each intersection. This study also integrates the ecological traffic concept into MARL traffic signal control, which aims to reduce traffic exhaust emissions. Finally, the proposed GCQN-TSC is experimentally validated both in a synthetic traffic grid and a real-world traffic network using the SUMO simulator. The experimental results show that GCQN-TSC outperforms other traffic signal control methods in almost all performance metrics, including average queue length and waiting time, as it can aggregate information acquired from collaborative agents and make network-level signal optimization decisions.

computer science, artificial intelligence
Attention Based Graph Covolution Networks for Intelligent Traffic Flow Analysis

Hongxin Zhang,Jiaxin Liu,Ying Tang,Gang Xiong

DOI: https://doi.org/10.1109/case48305.2020.9216966

2020-01-01

Abstract:Predicting traffic flow is one of the fundamental issue to smart cities. However it is still challenging in vehicular cyber-physical systems because of ever-increasing urban traffic data. Most previous models for traffic flow prediction suffers the problem with variance error depends on time and location. In this paper, a novel graph convolution network (GCN) approach is proposed for predicting citywide traffic flow. Our key idea is to introduce spatial and temporal attention to the GCN model to lessen the impact of urban data complexity. A framework of flow-based graph convolutional network is established to improve traffic flow prediction while investigating the spatial and temporal correlation of traffic flow. We further propose practical strategies that efficiently learn parameters of the model. Experimental results demonstrate that the proposed approach to traffic flow prediction outperforms state-of-the-art approaches.
Network Clustering-Based Multi-Agent Reinforcement Learning for Large-Scale Traffic Signal Control

Zhicheng Tao,Chao Li,Qinmin Yang

DOI: https://doi.org/10.1109/csis-iac60628.2023.10364020

2023-01-01

Abstract:Reinforcement learning (RL) has proven successful in the field of traffic signal control (TSC) within urban traffic networks. Moreover, multi-agent RL (MARL) achieves large-scale TSC by distributing the global control to each local RL agent. However, as the scale of joint TSC continues to expand, MARL faces greater convergence challenges. In this paper, a method combining network clustering and MARL is proposed to address this issue. We apply the Normalized Cut (Ncut) algorithm to effectively partition a large-scale traffic network, and then adopt MADQN to complete TSC tasks for each subnetwork. Meanwhile, a novel reward function based on impedance has been adopted. The results demonstrate that our method has high control performance and could effectively alleviate congestion in large-scale network under peak-hour traffic conditions.
A large-scale traffic signal control algorithm based on multi-layer graph deep reinforcement learning

Tao Wang,Zhipeng Zhu,Jing Zhang,Junfang Tian,Wenyi Zhang

DOI: https://doi.org/10.1016/j.trc.2024.104582

2024-05-01

Abstract:Due to its capability in handling complex urban intersection environments, deep reinforcement learning (DRL) has been widely applied in Adaptive Traffic Signal Control (ATSC). However, most existing algorithms are designed for specific road networks or traffic conditions, making it difficult to transfer them to new environments. Moreover, current graph-based algorithms do not fully capture the geometric and spatial features of intersections, leading to incomplete embedding of the agent's environment depiction. Additionally, the actions adopted by these algorithms are inherently based on fixed-cycle phases, limiting the flexibility of traffic signal control. To address the aforementioned issues, this paper proposes a Multi-layer Graph Mask Q-Learning (MGMQ) algorithm for multi-intersection ATSC to optimize traffic and reduce delay. Unlike previous graph-based algorithms, this paper introduces a method for computing multi-layer graphs, dividing the traffic environment into upper-level traffic network-layer graphs and lower-level intersection-layer graphs, and employs the graph attention algorithm and an improved GraphSAGE algorithm for computation. This method not only enables the generation of embedded state for intersections that include geometric and spatial features, but also allows the algorithm to adapt to different traffic conditions and road networks. Additionally, we introduce an action masking mechanism, allowing this algorithm can be adapted to different action spaces. As a result, the algorithm uses arbitrary signal phases as actions to achieve flexible traffic flow control, and can be directly applied to intersections with arbitrary geometry. The final test results demonstrate that a model trained solely on synthetic road networks can be directly transferred to other synthetic network configurations or real-world urban road networks, outperforming current state-of-the-art algorithms.

transportation science & technology
Adaptive Traffic Signal Control Based on Graph Neural Networks and Dynamic Entropy-Constrained Soft Actor–Critic

Xianguang Jia,Mengyi Guo,Yingying Lyu,Jie Qu,Dong Li,Fengxiang Guo

DOI: https://doi.org/10.3390/electronics13234794

IF: 2.9

2024-12-06

Electronics

Abstract:Traffic congestion remains a significant challenge in urban management, with traditional fixed-cycle traffic signal systems struggling to adapt to dynamic traffic conditions. This paper proposes an adaptive traffic signal control method based on a Graph Neural Network (GNN) and a dynamic entropy-constrained Soft Actor–Critic (DESAC) algorithm. The approach first extracts both global and local features of the traffic network using GNN and then utilizes the DESAC algorithm to optimize traffic signal control at both single and multi-intersection levels. Finally, a simulation environment is established on the CityFlow platform to evaluate the proposed method's performance through experiments involving single and twelve intersection scenarios. Simulation results on the CityFlow platform demonstrate that G-DESAC significantly improves traffic flow, reduces delays and queue lengths, and enhances intersection capacity compared to other algorithms. In single intersection scenarios, G-DESAC achieves a higher reward, reduced total delay time, minimized queue lengths, and improved throughput. In multi-intersection scenarios, G-DESAC maintains high rewards with stable and efficient optimization, outperforming DQN, SAC, Max-Pressure, and DDPG. This research highlights the potential of deep reinforcement learning (DRL) in urban traffic management and positions G-DESAC as a robust solution for practical traffic signal control applications, offering substantial improvements in traffic efficiency and congestion mitigation.

engineering, electrical & electronic,computer science, information systems,physics, applied
AGRCNet: communicate by attentional graph relations in multi-agent reinforcement learning for traffic signal control

Tinghuai Ma,Kexing Peng,Huan Rong,Yurong Qian

DOI: https://doi.org/10.1007/s00521-023-08875-5

2023-07-30

Neural Computing and Applications

Abstract:Traffic signal control (TSC) can be described as a multi-agent cooperative game. To realize cooperation, multi-agent reinforcement learning (MARL) is a significant approach, with communication being a core component. The large-scale traffic signals and the partially observable information in TSC pose a considerable challenge in finding the optimal joint control policy. This paper proposed a deep MARL model named attentional graph relations communications network (AGRCNet). Based on the Actor-Critic framework, AGRCNet designs a communication network to exchange observation information with agents to help obtain the optimal joint action, reducing the decision error caused by the partially observable condition. Specifically, through the communication network, the chain propagation of graph attention networks (GAT) and graph convolutional networks is used to expand the receptive domain of agents, improve communication efficiency and promote cooperative behavior. We simulate the traffic situation near the Nanjing Yangtze River Bridge in Simulation of Urban MObility. With a compound reward, our method performs best. Meanwhile, AGRCNet is applied to two abstract environments, and the results show that our approach can also adapt to dynamic agent relationships and is more efficient than comparison algorithms.

computer science, artificial intelligence
Scalable Traffic Signal Controls using Fog-Cloud Based Multiagent Reinforcement Learning

Paul,Sikai Chen,Runjia Du,Samuel Labi

DOI: https://doi.org/10.48550/arXiv.2110.05564

2021-10-12

Abstract:Optimizing traffic signal control (TSC) at intersections continues to pose a challenging problem, particularly for large-scale traffic networks. It has been shown in past research that it is feasible to optimize the operations of individual TSC systems or a small number of such systems. However, it has been computationally difficult to scale these solution approaches to large networks partly due to the curse of dimensionality that is encountered as the number of intersections increases. Fortunately, recent studies have recognized the potential of exploiting advancements in deep and reinforcement learning to address this problem, and some preliminary successes have been achieved in this regard. However, facilitating such intelligent solution approaches may require large amounts of infrastructural investments such as roadside units (RSUs) and drones in order to ensure thorough connectivity across all intersections in large networks, an investment that may be burdensome for agencies to undertake. As such, this study builds on recent work to present a scalable TSC model that may reduce the number of required enabling infrastructure. This is achieved using graph attention networks (GATs) to serve as the neural network for deep reinforcement learning, which aids in maintaining the graph topology of the traffic network while disregarding any irrelevant or unnecessary information. A case study is carried out to demonstrate the effectiveness of the proposed model, and the results show much promise. The overall research outcome suggests that by decomposing large networks using fog-nodes, the proposed fog-based graphic RL (FG-RL) model can be easily applied to scale into larger traffic networks.

Machine Learning,Artificial Intelligence
A DRL-based Multiagent Cooperative Control Framework for CAV Networks: a Graphic Convolution Q Network

Jiqian Dong,Sikai Chen,Paul Young Joun Ha,Yujie Li,Samuel Labi

DOI: https://doi.org/10.48550/arXiv.2010.05437

2020-10-12

Abstract:Connected Autonomous Vehicle (CAV) Network can be defined as a collection of CAVs operating at different locations on a multilane corridor, which provides a platform to facilitate the dissemination of operational information as well as control instructions. Cooperation is crucial in CAV operating systems since it can greatly enhance operation in terms of safety and mobility, and high-level cooperation between CAVs can be expected by jointly plan and control within CAV network. However, due to the highly dynamic and combinatory nature such as dynamic number of agents (CAVs) and exponentially growing joint action space in a multiagent driving task, achieving cooperative control is NP hard and cannot be governed by any simple rule-based methods. In addition, existing literature contains abundant information on autonomous driving's sensing technology and control logic but relatively little guidance on how to fuse the information acquired from collaborative sensing and build decision processor on top of fused information. In this paper, a novel Deep Reinforcement Learning (DRL) based approach combining Graphic Convolution Neural Network (GCN) and Deep Q Network (DQN), namely Graphic Convolution Q network (GCQ) is proposed as the information fusion module and decision processor. The proposed model can aggregate the information acquired from collaborative sensing and output safe and cooperative lane changing decisions for multiple CAVs so that individual intention can be satisfied even under a highly dynamic and partially observed mixed traffic. The proposed algorithm can be deployed on centralized control infrastructures such as road-side units (RSU) or cloud platforms to improve the CAV operation.

Artificial Intelligence,Systems and Control
Learning Decentralized Traffic Signal Controllers with Multi-Agent Graph Reinforcement Learning

Yao Zhang,Zhiwen Yu,Jun Zhang,Liang Wang,Tom H. Luan,Bin Guo,Chau Yuen

2023-11-07

Abstract:This paper considers optimal traffic signal control in smart cities, which has been taken as a complex networked system control problem. Given the interacting dynamics among traffic lights and road networks, attaining controller adaptivity and scalability stands out as a primary challenge. Capturing the spatial-temporal correlation among traffic lights under the framework of Multi-Agent Reinforcement Learning (MARL) is a promising solution. Nevertheless, existing MARL algorithms ignore effective information aggregation which is fundamental for improving the learning capacity of decentralized agents. In this paper, we design a new decentralized control architecture with improved environmental observability to capture the spatial-temporal correlation. Specifically, we first develop a topology-aware information aggregation strategy to extract correlation-related information from unstructured data gathered in the road network. Particularly, we transfer the road network topology into a graph shift operator by forming a diffusion process on the topology, which subsequently facilitates the construction of graph signals. A diffusion convolution module is developed, forming a new MARL algorithm, which endows agents with the capabilities of graph learning. Extensive experiments based on both synthetic and real-world datasets verify that our proposal outperforms existing decentralized algorithms.

Machine Learning,Artificial Intelligence,Systems and Control
Graph neural network and reinforcement learning for multi‐agent cooperative control of connected autonomous vehicles

Sikai Chen,Jiqian Dong,Paul Ha,Yujie Li,Samuel Labi,Paul (Young Joun) Ha

DOI: https://doi.org/10.1111/mice.12702

2021-06-10

Abstract:<p>A connected autonomous vehicle (CAV) network can be defined as a set of connected vehicles including CAVs that operate on a specific spatial scope that may be a road network, corridor, or segment. The spatial scope constitutes an environment where traffic information is shared and instructions are issued for controlling the CAVs movements. Within such a spatial scope, high-level cooperation among CAVs fostered by joint planning and control of their movements can greatly enhance the safety and mobility performance of their operations. Unfortunately, the highly combinatory and volatile nature of CAV networks due to the dynamic number of agents (vehicles) and the fast-growing joint action space associated with multi-agent driving tasks pose difficultly in achieving cooperative control. The problem is NP-hard and cannot be efficiently resolved using rule-based control techniques. Also, there is a great deal of information in the literature regarding sensing technologies and control logic in CAV operations but relatively little information on the integration of information from collaborative sensing and connectivity sources. Therefore, we present a novel deep reinforcement learning-based algorithm that combines graphic convolution neural network with deep Q-network to form an innovative graphic convolution Q network that serves as the information fusion module and decision processor. In this study, the spatial scope we consider for the CAV network is a multi-lane road corridor. We demonstrate the proposed control algorithm using the application context of freeway lane-changing at the approaches to an exit ramp. For purposes of comparison, the proposed model is evaluated vis-à-vis traditional rule-based and long short-term memory-based fusion models. The results suggest that the proposed model is capable of aggregating information received from sensing and connectivity sources and prescribing efficient operative lane-change decisions for multiple CAVs, in a manner that enhances safety and mobility. That way, the operational intentions of individual CAVs can be fulfilled even in partially observed and highly dynamic mixed traffic streams. The paper presents experimental evidence to demonstrate that the proposed algorithm can significantly enhance CAV operations. The proposed algorithm can be deployed at roadside units or cloud platforms or other centralized control facilities.</p>

transportation science & technology,construction & building technology,computer science, interdisciplinary applications,engineering, civil
Multi-agent Deep Reinforcement Learning collaborative Traffic Signal Control method considering intersection heterogeneity

Yiming Bie,Yuting Ji,Dongfang Ma

DOI: https://doi.org/10.1016/j.trc.2024.104663

IF: 9.022

2024-05-25

Transportation Research Part C Emerging Technologies

Abstract:Traffic Signal Control (TSC) plays a crucial role in mitigating congestion. The extensive integration of Deep Reinforcement Learning (DRL) into TSC, fueled by the advancements in artificial intelligence, has yielded remarkable efficacy. Nevertheless, prevailing DRL models are typically developed within homogeneous road networks, characterized by identical intersection types and relatively balanced traffic flow patterns. This predominant approach often results in an oversight of the impact of intersection heterogeneity, encompassing variations in intersection geometric structure, traffic demands, signal timing design, and other pertinent factors, on action selections by researchers. This paper addresses the aforementioned gap by introducing a value-decomposition-based spatiotemporal graph attention multi-agent DRL model (MARL_SGAT) that expressly accommodates intersection heterogeneity. Specifically, a heterogeneous correlation index is designed using road network structural parameters and serves to formulate a heterogeneous reward function, enabling the quantification of action execution improvements in diverse road network environments. To address the intricate spatiotemporal features of traffic flows within the heterogeneous road network, the model strategically selects and incorporates these features during the formulation by leveraging a spatiotemporal graph attention network. On these bases, double dual networks are introduced to convert the individual-global-max constraint of action selection into a value range constraint of the action advantage function, thereby facilitating model learning. Simulation results showcase the superior performance of the MARL_SGAT algorithm when compared to seven baseline algorithms. Specifically, the algorithm manifests in reduced average vehicle delays, decreased number of stops, and elevated travel speeds within the controlled road network. These advantages are particularly conspicuous in heterogeneous road network environments.

transportation science & technology
Enhancing traffic signal control with composite deep intelligence

Zhongnan Zhao,Kun Wang,Yue Wang,Xiaoliang Liang

DOI: https://doi.org/10.1016/j.eswa.2023.123020

IF: 8.5

2024-06-01

Expert Systems with Applications

Abstract:Traffic signal control has always been a hot topic in the field of intelligent transportation. With the increasing complexity of urban traffic conditions due to urbanization, how to develop effective scheduling strategies to adapt to the changing traffic demands has become a key problem in current intelligent transportation. In light of this, this paper focuses on the traffic signal control problem at intersections and proposes a composite intelligent traffic signal control model based on heterogeneous graph neural networks with dual attention mechanisms and deep reinforcement learning. For the first time, the model incorporates the dual attention mechanism in graph neural networks into the traffic signal control, integrating graph neural networks with deep reinforcement learning techniques and traffic intersection scenarios. This allows for the construction of traffic condition models and the scheduling control of traffic resources, catering to the perception and decision-making needs in complex traffic environments. Firstly, the graph relationship representation of intersection resources is established, constructing the graph information structure for traffic flow and signal states. Then, a heterogeneous graph neural network is designed, incorporating both node-level and semantic-level dual attention mechanisms to characterize the traffic state and explore the relationships, enabling the extraction of explicit and implicit information in traffic intersections. Lastly, a deep reinforcement learning algorithm that combines Double Deep Q-Network (DDQN) and Dueling DQN is implemented to improve the algorithm's generalization and execution efficiency, enhancing the adaptability and stability of traffic signal scheduling in complex environments. Simulation tests are conducted on the SUMO simulation platform using real-world application datasets. Compared to four other similar traffic control model, the proposed model demonstrates performance advantages of more than 13% in terms of average reward, average delay, queue length, and waiting time. This validates the effectiveness of the proposed model.

computer science, artificial intelligence,engineering, electrical & electronic,operations research & management science
Cooperative Multi-Agent Actor–critic Control of Traffic Network Flow Based on Edge Computing

Yongnan Zhang,Yonghua Zhou,Huapu Lu,Hamido Fujita

DOI: https://doi.org/10.1016/j.future.2021.04.018

IF: 7.307

2021-01-01

Future Generation Computer Systems

Abstract:Most of the existing traffic signal control strategies are hard to satisfy the real-time requirements of traffic big data analysis, knowledge reasoning and decision making for sophisticated traffic dynamics and heterogeneous intersection structures in the context of Internet of Vehicles (IoV). In this paper, we attempt to propose a cooperative multi-agent actor–critic (CMAC) deep reinforcement learning (DRL) approach with value decomposition based on edge computing architecture. The intuition behind CMAC is to decompose the global actor–critic learning tasks into several local actor–critic sub-problems with respect to each intersection. Each agent searches the local optimal decision by actor–critic network that takes the discrete state encoding about several consecutive frames of image-like traffic states as the inputs of the network. Among them, the green ratio output strategy considering multiple constraints is formulated in the output layer of the actor network, so that the continuous control of traffic signals using multi-agent DRL (MADRL) can be realized. Furthermore, a cooperative mechanism that considers contribution weight distributions of local agents to the global traffic pattern is proposed to coordinate multiple local agents to evolve toward global optimization. Especially, some parallel training tasks of CMAC with a large number of computing loads are deployed on the cloud side in the edge computing architecture to accelerate learning and reconstructing knowledge. The well-trained multi-agent model is downloaded from the cloud side into the edge side for real-time decision making of traffic network flow adaptive control. Simulation results with regard to a realistic traffic network demonstrate that the proposed CMAC approach under edge computing architecture outperforms the value-decomposition based multi-agent actor–critic (VMAC), independent multi-agent actor–critic (IMAC), and the fixed timing control (FTC) in terms of alleviating traffic congestion.
Feudal Multi-Agent Reinforcement Learning with Adaptive Network Partition for Traffic Signal Control

Jinming Ma,Feng Wu

DOI: https://doi.org/10.48550/arXiv.2205.13836

2022-05-27

Multiagent Systems

Abstract:Multi-agent reinforcement learning (MARL) has been applied and shown great potential in multi-intersections traffic signal control, where multiple agents, one for each intersection, must cooperate together to optimize traffic flow. To encourage global cooperation, previous work partitions the traffic network into several regions and learns policies for agents in a feudal structure. However, static network partition fails to adapt to dynamic traffic flow, which will changes frequently over time. To address this, we propose a novel feudal MARL approach with adaptive network partition. Specifically, we first partition the network into several regions according to the traffic flow. To do this, we propose two approaches: one is directly to use graph neural network (GNN) to generate the network partition, and the other is to use Monte-Carlo tree search (MCTS) to find the best partition with criteria computed by GNN. Then, we design a variant of Qmix using GNN to handle various dimensions of input, given by the dynamic network partition. Finally, we use a feudal hierarchy to manage agents in each partition and promote global cooperation. By doing so, agents are able to adapt to the traffic flow as required in practice. We empirically evaluate our method both in a synthetic traffic grid and real-world traffic networks of three cities, widely used in the literature. Our experimental results confirm that our method can achieve better performance, in terms of average travel time and queue length, than several leading methods for traffic signal control.
Decentralized Deep Reinforcement Learning for Network Level Traffic Signal Control

Jin Guo

DOI: https://doi.org/10.48550/arXiv.2007.03433

2020-07-18

Abstract:In this thesis, I propose a family of fully decentralized deep multi-agent reinforcement learning (MARL) algorithms to achieve high, real-time performance in network-level traffic signal control. In this approach, each intersection is modeled as an agent that plays a Markovian Game against the other intersection nodes in a traffic signal network modeled as an undirected graph, to approach the optimal reduction in delay. Following Partially Observable Markov Decision Processes (POMDPs), there are 3 levels of communication schemes between adjacent learning agents: independent deep Q-leaning (IDQL), shared states reinforcement learning (S2RL) and a shared states & rewards version of S2RL--S2R2L. In these 3 variants of decentralized MARL schemes, individual agent trains its local deep Q network (DQN) separately, enhanced by convergence-guaranteed techniques like double DQN, prioritized experience replay, multi-step bootstrapping, etc. To test the performance of the proposed three MARL algorithms, a SUMO-based simulation platform is developed to mimic the traffic evolution of the real world. Fed with random traffic demand between permitted OD pairs, a 4x4 Manhattan-style grid network is set up as the testbed, two different vehicle arrival rates are generated for model training and testing. The experiment results show that S2R2L has a quicker convergence rate and better convergent performance than IDQL and S2RL in the training process. Moreover, three MARL schemes all reveal exceptional generalization abilities. Their testing results surpass the benchmark Max Pressure (MP) algorithm, under the criteria of average vehicle delay, network-level queue length and fuel consumption rate. Notably, S2R2L has the best testing performance of reducing 34.55% traffic delay and dissipating 10.91% queue length compared with MP.

Machine Learning,Artificial Intelligence,Signal Processing
Towards Multi-agent Reinforcement Learning based Traffic Signal Control through Spatio-temporal Hypergraphs

Kang Wang,Zhishu Shen,Zhen Lei,Tiehua Zhang

2024-04-17

Abstract:Traffic signal control systems (TSCSs) are integral to intelligent traffic management, fostering efficient vehicle flow. Traditional approaches often simplify road networks into standard graphs, which results in a failure to consider the dynamic nature of traffic data at neighboring intersections, thereby neglecting higher-order interconnections necessary for real-time control. To address this, we propose a novel TSCS framework to realize intelligent traffic control. This framework collaborates with multiple neighboring edge computing servers to collect traffic information across the road network. To elevate the efficiency of traffic signal control, we have crafted a multi-agent soft actor-critic (MA-SAC) reinforcement learning algorithm. Within this algorithm, individual agents are deployed at each intersection with a mandate to optimize traffic flow across the entire road network collectively. Furthermore, we introduce hypergraph learning into the critic network of MA-SAC to enable the spatio-temporal interactions from multiple intersections in the road network. This method fuses hypergraph and spatio-temporal graph structures to encode traffic data and capture the complex spatial and temporal correlations between multiple intersections. Our empirical evaluation, tested on varied datasets, demonstrates the superiority of our framework in minimizing average vehicle travel times and sustaining high-throughput performance. This work facilitates the development of more intelligent and reactive urban traffic management solutions.

Multiagent Systems,Artificial Intelligence
Combat Urban Congestion via Collaboration: Heterogeneous GNN-based MARL for Coordinated Platooning and Traffic Signal Control

Xianyue Peng,Hang Gao,Hao Wang,H. Michael Zhang

2023-10-17

Abstract:Over the years, reinforcement learning has emerged as a popular approach to develop signal control and vehicle platooning strategies either independently or in a hierarchical way. However, jointly controlling both in real-time to alleviate traffic congestion presents new challenges, such as the inherent physical and behavioral heterogeneity between signal control and platooning, as well as coordination between them. This paper proposes an innovative solution to tackle these challenges based on heterogeneous graph multi-agent reinforcement learning and traffic theories. Our approach involves: 1) designing platoon and signal control as distinct reinforcement learning agents with their own set of observations, actions, and reward functions to optimize traffic flow; 2) designing coordination by incorporating graph neural networks within multi-agent reinforcement learning to facilitate seamless information exchange among agents on a regional scale. We evaluate our approach through SUMO simulation, which shows a convergent result in terms of various transportation metrics and better performance over sole signal or platooning control.

Machine Learning,Multiagent Systems
Leveraging the Capabilities of Connected and Autonomous Vehicles and Multi-Agent Reinforcement Learning to Mitigate Highway Bottleneck Congestion

Paul Young Joun Ha,Sikai Chen,Jiqian Dong,Runjia Du,Yujie Li,Samuel Labi

DOI: https://doi.org/10.48550/arXiv.2010.05436

2020-10-12

Abstract:Active Traffic Management strategies are often adopted in real-time to address such sudden flow breakdowns. When queuing is imminent, Speed Harmonization (SH), which adjusts speeds in upstream traffic to mitigate traffic showckwaves downstream, can be applied. However, because SH depends on driver awareness and compliance, it may not always be effective in mitigating congestion. The use of multiagent reinforcement learning for collaborative learning, is a promising solution to this challenge. By incorporating this technique in the control algorithms of connected and autonomous vehicle (CAV), it may be possible to train the CAVs to make joint decisions that can mitigate highway bottleneck congestion without human driver compliance to altered speed limits. In this regard, we present an RL-based multi-agent CAV control model to operate in mixed traffic (both CAVs and human-driven vehicles (HDVs)). The results suggest that even at CAV percent share of corridor traffic as low as 10%, CAVs can significantly mitigate bottlenecks in highway traffic. Another objective was to assess the efficacy of the RL-based controller vis-à-vis that of the rule-based controller. In addressing this objective, we duly recognize that one of the main challenges of RL-based CAV controllers is the variety and complexity of inputs that exist in the real world, such as the information provided to the CAV by other connected entities and sensed information. These translate as dynamic length inputs which are difficult to process and learn from. For this reason, we propose the use of Graphical Convolution Networks (GCN), a specific RL technique, to preserve information network topology and corresponding dynamic length inputs. We then use this, combined with Deep Deterministic Policy Gradient (DDPG), to carry out multi-agent training for congestion mitigation using the CAV controllers.

Machine Learning,Systems and Control
Traffic Signal Control Optimization Based on Neural Network in the Framework of Model Predictive Control

Dapeng Tang,Yuzhou Duan

DOI: https://doi.org/10.3390/act13070251

IF: 2.6

2024-07-01

Actuators

Abstract:To improve the effectiveness of model predictive control (MPC) in dynamic traffic signal control strategies, it has been combined with graph convolutional networks (GCNs) and deep reinforcement learning (DRL) technologies. In this study, a neural-network-based traffic signal control optimization method under the MPC framework is proposed. A dynamic correlation matrix is introduced in the predictive model to adapt to the dynamic changes in correlations between nodes over time. The signal control optimization strategy is solved using DRL, where the agent explores the optimal control strategy based on pre-set constraints in the future road environment. The geometric structure and traffic flow data of a real intersection were selected as the simulation validation environment, and a joint simulation was conducted using Python and SUMO. The experimental results indicate that in low-traffic scenarios, the queue length is reduced by more than 2 vehicles compared to the selected comparison methods; in high-traffic scenarios, the queue length is reduced by an average of 17 vehicles. Under the actual traffic data of the intersection, the average speed is increased by 6.4% compared to the fixed timing method; compared to the inductive signal control method, it increases from 9.76 m/s to 11.69 m/s, an improvement of 19.7%, effectively enhancing the intersection signal control performance.

engineering, mechanical,instruments & instrumentation

[The vaccines of the future].

Graph cooperation deep reinforcement learning for ecological urban traffic signal control

Attention Based Graph Covolution Networks for Intelligent Traffic Flow Analysis

Network Clustering-Based Multi-Agent Reinforcement Learning for Large-Scale Traffic Signal Control

A large-scale traffic signal control algorithm based on multi-layer graph deep reinforcement learning

Adaptive Traffic Signal Control Based on Graph Neural Networks and Dynamic Entropy-Constrained Soft Actor–Critic

AGRCNet: communicate by attentional graph relations in multi-agent reinforcement learning for traffic signal control

Scalable Traffic Signal Controls using Fog-Cloud Based Multiagent Reinforcement Learning

A DRL-based Multiagent Cooperative Control Framework for CAV Networks: a Graphic Convolution Q Network

Learning Decentralized Traffic Signal Controllers with Multi-Agent Graph Reinforcement Learning

Graph neural network and reinforcement learning for multi‐agent cooperative control of connected autonomous vehicles

Multi-agent Deep Reinforcement Learning collaborative Traffic Signal Control method considering intersection heterogeneity

Enhancing traffic signal control with composite deep intelligence

Cooperative Multi-Agent Actor–critic Control of Traffic Network Flow Based on Edge Computing

Feudal Multi-Agent Reinforcement Learning with Adaptive Network Partition for Traffic Signal Control

Decentralized Deep Reinforcement Learning for Network Level Traffic Signal Control

Towards Multi-agent Reinforcement Learning based Traffic Signal Control through Spatio-temporal Hypergraphs

Combat Urban Congestion via Collaboration: Heterogeneous GNN-based MARL for Coordinated Platooning and Traffic Signal Control

Leveraging the Capabilities of Connected and Autonomous Vehicles and Multi-Agent Reinforcement Learning to Mitigate Highway Bottleneck Congestion

Traffic Signal Control Optimization Based on Neural Network in the Framework of Model Predictive Control