Abstract:With the development of unmanned aerial vehicle (UAV) and artificial intelligence (AI) technology, Intelligent UAV will be widely used in future autonomous aerial combat. Previous researches on autonomous aerial combat within visual range (WVR) have limitations due to simplifying assumptions, limited robustness, and ignoring sensor errors. In this paper, in order to consider the error of the aircraft sensors, we model the aerial combat WVR as a state-adversarial Markov decision process (SA-MDP), which introduce the small adversarial perturbations on state observations and these perturbations do not alter the environment directly, but can mislead the agent into making suboptimal decisions. Meanwhile, we propose a novel autonomous aerial combat maneuver strategy generation algorithm with high-performance and high-robustness based on state-adversarial deep deterministic policy gradient algorithm (SA-DDPG), which add a robustness regularizers related to an upper bound on performance loss at the actor-network. At the same time, a reward shaping method based on maximum entropy (MaxEnt) inverse reinforcement learning algorithm (IRL) is proposed to improve the aerial combat strategy generation algorithm’s efficiency. Finally, the efficiency of the aerial combat strategy generation algorithm and the performance and robustness of the resulting aerial combat strategy is verified by simulation experiments. Our main contributions are three-fold. First, to introduce the observation errors of UAV, we are modeling air combat as SA-MDP. Second, to make the strategy network of air combat maneuver more robust in the presence of observation errors, we introduce regularizers into the policy gradient. Third, to solve the problem that air combat’s reward function is too sparse, we use MaxEnt IRL to design a shaping reward to accelerate the convergence of SA-DDPG.

The Development of Rule-based AI Engagement Model for Air-to-Air Combat Simulation

Manual-Based Automated Maneuvering Decisions for Air-to-Air Combat

Explainable Basic-Fighter-Maneuver Decision Support Scheme for Piloting Within-Visual-Range Air Combat

Maneuver Decision of UAV in Short-Range Air Combat Based on Deep Reinforcement Learning

Deep Reinforcement-Learning-Based Air-Combat-Maneuver Generation Framework

H3E: Learning air combat with a three-level hierarchical framework embedding expert knowledge

Knowledge-based and data-driven behavioral modeling techniques in engagement simulation

Engagement Decision Support for Beyond Visual Range Air Combat

Hierarchical Reinforcement Learning from Competitive Self-play for Dual-aircraft formation air combat

Multi-intent autonomous decision-making for air combat with deep reinforcement learning

A Multi-UCAV cooperative occupation method based on weapon engagement zones for beyond-visual-range air combat

UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning

A survey of air combat behavior modeling using machine learning

Cross coordination of behavior clone and reinforcement learning for autonomous within-visual-range air combat

An Air Combat Decision Learning System Based on a Brain-Like Cognitive Mechanism

Autonomous Agent for Beyond Visual Range Air Combat: A Deep Reinforcement Learning Approach

Game of Drones: Intelligent Online Decision Making of Multi-UAV Confrontation

BVR Gym: A Reinforcement Learning Environment for Beyond-Visual-Range Air Combat

UAV Autonomous Aerial Combat Maneuver Strategy Generation with Observation Error Based on State-Adversarial Deep Deterministic Policy Gradient and Inverse Reinforcement Learning

Air Combat Maneuver Decision Based on Deep Reinforcement Learning and Game Theory

High-Fidelity Decision-Making and Simulation for Cooperative Autonomous Air Combat Considering the Effect of Flight Controller