Abstract:Unmanned aerial vehicles (UAV), also known as drones have been used for a variety of reasons and the commercial drone market growth is expected to reach remarkable levels in the near future. However, some drone users can mistakenly or intentionally fly into flight paths at major airports, flying too close to commercial aircraft or invading people's privacy. In order to prevent these unwanted events, counter-drone technology is needed to eliminate threats from drones and hopefully they can be integrated into the skies safely. There are various counter-drone methods available in the industry. However, a counter-drone system supported by an artificial intelligence (AI) method can be an efficient way to fight against drones instead of human intervention. In this paper, a deep reinforcement learning (DRL) method has been proposed to counter a drone in a 3D space by using another drone. In a 2D space it is already shown that the deep reinforcement learning method is an effective way to counter a drone. However, countering a drone in a 3D space with another drone is a very challenging task considering the time required to train and avoid obstacles at the same time. A Deep Q-Network (DQN) algorithm with dueling network architecture and prioritized experience replay is presented to catch another drone in the environment provided by an Airsim simulator. The models have been trained and tested with different scenarios to analyze the learning progress of the drone. Experiences from previous training are also transferred before starting a new training by pre-processing the previous experiences and eliminating those considered as bad experiences. The results show that the best models are obtained with transfer learning and the drone learning progress has been increased dramatically. Additionally, an algorithm which combines imitation learning and reinforcement learning is implemented to catch the target drone. In this algorithm, called deep q-learning from demonstrations (DQfD), expert demonstrations data and self-generated data by the agent are sampled and the agent continues learning without overwriting the demonstration data. The main advantage of this algorithm is to accelerate the learning process even if there is a small amount of demonstration data.

Air-Combat Strategy Using Deep Q-Learning

Model-free Maneuvering Control of Fixed-Wing UAVs Based on Deep Reinforcement Learning

A Deep Reinforcement Learning Based Intelligent Decision Method for UCAV Air Combat

Air Combat Maneuver Decision Based on Deep Reinforcement Learning and Game Theory

Maneuver Decision of UAV in Short-Range Air Combat Based on Deep Reinforcement Learning

Autonomous Maneuver Decision of UCAV Air Combat Based on Double Deep Q Network Algorithm and Stochastic Game Theory

Deep Reinforcement-Learning-Based Air-Combat-Maneuver Generation Framework

Deep Reinforcement Learning With Application to Air Confrontation Intelligent Decision-Making of Manned/Unmanned Aerial Vehicle Cooperative System

Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs

Autonomous Decision-Making Method for Combat Mission of UAV based on Deep Reinforcement Learning

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

Application of Deep Reinforcement Learning to Defense and Intrusion Strategies Using Unmanned Aerial Vehicles in a Versus Game

Application of Deep Reinforcement Learning in UAVs: A Review

A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat

Autonomous maneuver decision-making for a UCAV in short-range aerial combat based on an MS-DDQN algorithm

Multi-intent autonomous decision-making for air combat with deep reinforcement learning

Autonomous UAV maneuvering decisions by refining opponent strategies

Deep Reinforcement Learning for UAV Intelligent Mission Planning

Towards Real-Time Path Planning through Deep Reinforcement Learning for a UAV in Dynamic Environments

Countering a Drone in a 3D Space: Analyzing Deep Reinforcement Learning Methods

Research on Deep Q-Network Hybridization with Extended Kalman Filter in Maneuvering Decision of Unmanned Combat Aerial Vehicles