Abstract:With the development of unmanned aerial vehicle (UAV) and artificial intelligence (AI) technology, Intelligent UAV will be widely used in future autonomous aerial combat. Previous researches on autonomous aerial combat within visual range (WVR) have limitations due to simplifying assumptions, limited robustness, and ignoring sensor errors. In this paper, in order to consider the error of the aircraft sensors, we model the aerial combat WVR as a state-adversarial Markov decision process (SA-MDP), which introduce the small adversarial perturbations on state observations and these perturbations do not alter the environment directly, but can mislead the agent into making suboptimal decisions. Meanwhile, we propose a novel autonomous aerial combat maneuver strategy generation algorithm with high-performance and high-robustness based on state-adversarial deep deterministic policy gradient algorithm (SA-DDPG), which add a robustness regularizers related to an upper bound on performance loss at the actor-network. At the same time, a reward shaping method based on maximum entropy (MaxEnt) inverse reinforcement learning algorithm (IRL) is proposed to improve the aerial combat strategy generation algorithm’s efficiency. Finally, the efficiency of the aerial combat strategy generation algorithm and the performance and robustness of the resulting aerial combat strategy is verified by simulation experiments. Our main contributions are three-fold. First, to introduce the observation errors of UAV, we are modeling air combat as SA-MDP. Second, to make the strategy network of air combat maneuver more robust in the presence of observation errors, we introduce regularizers into the policy gradient. Third, to solve the problem that air combat’s reward function is too sparse, we use MaxEnt IRL to design a shaping reward to accelerate the convergence of SA-DDPG.

UAV Air Combat Algorithm Based on Bayesian Probability Model

UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning

An Unmanned Air Combat System Based on Swarm Intelligence

Multi-UAV Cooperative Swarm Algorithm in Air Combat Based on Predictive Game Tree

Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning

Maneuver Decision of UAV in Short-Range Air Combat Based on Deep Reinforcement Learning

Modeling of UAV's Dynamic Search and Its Realization Algorithm

Research on Autonomous Maneuvering Decision of UCAV based on Approximate Dynamic Programming

UAV Autonomous Aerial Combat Maneuver Strategy Generation with Observation Error Based on State-Adversarial Deep Deterministic Policy Gradient and Inverse Reinforcement Learning

Autonomous Maneuver Decision of UCAV Air Combat Based on Double Deep Q Network Algorithm and Stochastic Game Theory

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

UCAV Air Combat Maneuver Decisions Based on a Proximal Policy Optimization Algorithm with Situation Reward Shaping

UAV Cooperative Air Combat Maneuvering Confrontation Based on Multi-agent Reinforcement Learning

Enhancing multi-UAV air combat decision making via hierarchical reinforcement learning

An Air Combat Decision Learning System Based on a Brain-Like Cognitive Mechanism

Research on UAV flight attitude control system based on simulink and genetic algorithm

Game of Drones: Intelligent Online Decision Making of Multi-UAV Confrontation

Cooperative control of UAV based on Multi-Agent System

High-Fidelity Decision-Making and Simulation for Cooperative Autonomous Air Combat Considering the Effect of Flight Controller

Robust Optimization based Air Combat Game Decision-making of Multi-UAV with Uncertain Information

Predictive air combat decision model with segmented reward allocation