Abstract:The unmanned aerial vehicle (UAV) has been applied in unmanned air combat because of its flexibility and practicality. The short-range air combat situation is rapidly changing, and the UAV has to make the autonomous maneuver decision as quickly as possible. In this paper, a type of short-range air combat maneuver decision method based on deep reinforcement learning is proposed. Firstly, the combat environment, including UAV motion model and the position and velocity relationships, is described. On this basic, the combat process is established. Secondly, some improved points based on proximal policy optimization (PPO) are proposed to enhance the maneuver decision-making ability. The gate recurrent unit (GRU) can help PPO make decisions with continuous timestep data. The actor network's input is the observation of UAV, however, the input of the critic network, named state, includes the blood values which cannot be observed directly. In addition, the action space with 15 basic actions and well-designed reward function are proposed to combine the air combat environment and PPO. In particular, the reward function is divided into dense reward, event reward and end-game reward to ensure the training feasibility. The training process is composed of three phases to shorten the training time. Finally, the designed maneuver decision method is verified through the ablation study and confrontment tests. The results show that the UAV with the proposed maneuver decision method can obtain an effective action policy to make a more flexible decision in air combat.

Hierarchical Reinforcement Learning for UAV-PE Game With Alternative Delay Update Method

Hierarchical Decision and Control for Continuous Multitarget Problem: Policy Evaluation with Action Delay

Model-free Maneuvering Control of Fixed-Wing UAVs Based on Deep Reinforcement Learning

Autonomous obstacle avoidance of UAV based on deep reinforcement learning

Autonomous Decision Making for UAV Cooperative Pursuit-Evasion Game with Reinforcement Learning

Deep Reinforcement Learning With Application to Air Confrontation Intelligent Decision-Making of Manned/Unmanned Aerial Vehicle Cooperative System

A Hierarchical Reinforcement Learning Algorithm Based on Attention Mechanism for UAV Autonomous Navigation

Adaptive collision-free control for UAVs with discrete-time system based on reinforcement learning

Responsive Regulation of Dynamic UAV Communication Networks Based on Deep Reinforcement Learning

Path Planning of Unmanned Aerial Vehicle in Complex Environments Based on State-Detection Twin Delayed Deep Deterministic Policy Gradient

Autonomous maneuver decision-making for a UCAV in short-range aerial combat based on an MS-DDQN algorithm

Heuristic Function Negotiation For Markov Decision Process And Its Application In Uav Simulation

Intelligent Maneuver Strategy for a Hypersonic Pursuit-Evasion Game Based on Deep Reinforcement Learning

UAV Obstacle Avoidance by Human-in-the-Loop Reinforcement in Arbitrary 3D Environment

Game of Drones: Multi-UAV Pursuit-Evasion Game With Online Motion Planning by Deep Reinforcement Learning

Deep reinforcement learning for unmanned aerial vehicles cluster task allocation

Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs

Pursuit and evasion game between UVAs based on multi-agent reinforcement learning

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

Reinforcement learning-based formation-surrounding control for multiple quadrotor UAVs pursuit-evasion games

Reinforcement Learning in Multiple-UAV Networks: Deployment and Movement Design