Abstract:Increasing traffic demands, higher levels of automation, and communication enhancements provide novel design opportunities for future air traffic controllers (ATCs). This article presents a novel deep reinforcement learning (DRL) controller to aid conflict resolution for autonomous free flight. Although DRL has achieved important advancements in this field, the existing works pay little attention to the explainability and safety issues related to DRL controllers, particularly the safety under adversarial attacks. To address those two issues, we design a fully explainable DRL framework wherein we: 1) decompose the coupled Q value learning model into a safety-awareness and efficiency (reach the target) one; and 2) use information from surrounding intruders as inputs, eliminating the needs of central controllers. In our simulated experiments, we show that by decoupling the safety-awareness and efficiency, we can exceed performance on free flight control tasks while dramatically improving explainability on practical. In addition, the safety Q learning module provides rich information about the safety situation of environments. To study the safety under adversarial attacks, we additionally propose an adversarial attack strategy that can impose both safety-oriented and efficiency-oriented attacks. The adversarial aims to minimize safety/efficiency by only attacking the agent at a few time steps. In the experiments, our attack strategy increases as many collisions as the uniform attack (i.e., attacking at every time step) by only attacking the agent four times less often, which provide insights into the capabilities and restrictions of the DRL in future ATC designs. The source code is publicly available at <a class="link-external link-https" href="https://github.com/WLeiiiii/Gym-ATC-Attack-Project" rel="external noopener nofollow">this https URL</a>.

Review of Deep Reinforcement Learning Approaches for Conflict Resolution in Air Traffic Control

Enhancing Air Traffic Control: A Transparent Deep Reinforcement Learning Framework for Autonomous Conflict Resolution

Study on the resolution of multi-aircraft flight conflicts based on an IDQN

Application of Deep Reinforcement Learning in UAVs: A Review

General real-time three-dimensional multi-aircraft conflict resolution method using multi-agent reinforcement learning

Deep Reinforcement Learning With Application to Air Confrontation Intelligent Decision-Making of Manned/Unmanned Aerial Vehicle Cooperative System

Autonomous air traffic separation assurance through machine learning

Solving the Problem of Detecting and Resolving Conflicts in Airspace Using Deep Reinforcement Learning

Leveraging Deep Reinforcement Learning for Traffic Engineering: A Survey

Explainable and Safe Reinforcement Learning for Autonomous Air Mobility

Deep reinforcement learning-based air combat maneuver decision-making: literature review, implementation tutorial and future direction

Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions

A Deep Ensemble Multi-Agent Reinforcement Learning Approach for Air Traffic Control

Deep Reinforcement Learning and Transportation Research: A Comprehensive Review

Deep Reinforcement Learning in Autonomous Car Path Planning and Control: A Survey

General multi-agent reinforcement learning integrating adaptive manoeuvre strategy for real-time multi-aircraft conflict resolution

Analysis of the impact of traffic density on training of reinforcement learning based conflict resolution methods for drones

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

A Survey on Reinforcement Learning in Aviation Applications

Leveraging reinforcement learning for dynamic traffic control: A survey and challenges for field implementation

A Comparative Analysis of Deep Reinforcement Learning-enabled Freeway Decision-making for Automated Vehicles