Abstract:Autonomous decision-making in unmanned aerial vehicle (UVA) confrontations presents challenges in making optimal strategy. Therefore, deep reinforcement learning (DRL) has been adopted to address these issues. However, existing DRL decision-making models suffer from poor situational awareness and inability to distinguish between different intentions. Therefore, a multi-intent autonomous decision-making is proposed in this paper. First, three typical intentions are designed comprising head-on attacking, pursuing and fleeing to derive decision models representing different intentions. Reinforcement learning based air combat game model is constructed with different intentions, which contains designing reward functions for intentions to deal with the problem of sparse rewards. Then, we propose the Temporal Proximal Policy Optimization (T-PPO) algorithm, which optimizes the Proximal Policy Optimization algorithm by integrating the long short-term memory network and feedforward neural network. This algorithm extracts the historical temporal information to enhance situational awareness. In addition, a basic-confrontation progressive training method is proposed to provide intention guidance and increase training diversity, which can improve learning efficiency and intelligent decision-making capability. Finally, experiments in our constructed UAV confrontation environment demonstrate that the proposed intentional decision models exhibit good performance in stability and learning efficiency, achieving high rewards, win rates, and low steps. Specifically, our autonomous decision-making increases win rate by 26% when head-on attacking and learning efficiency by 50% when pursuing. It is further proof of the potential and value of our multi-intent autonomous decision-making applications.

Air Combat Maneuver Decision Based on Deep Reinforcement Learning and Game Theory

Model-free Maneuvering Control of Fixed-Wing UAVs Based on Deep Reinforcement Learning

Maneuver Decision of UAV in Short-Range Air Combat Based on Deep Reinforcement Learning

UAV Countermeasure Maneuver Decision Based on Deep Reinforcement Learning

Autonomous Maneuver Decision of UCAV Air Combat Based on Double Deep Q Network Algorithm and Stochastic Game Theory

Manoeuvre Decision‐making of Unmanned Aerial Vehicles in Air Combat Based on an Expert Actor‐based Soft Actor Critic Algorithm

Air Combat Maneuver Decision Method Based on A3C Deep Reinforcement Learning

Improving Maneuver Strategy in Air Combat by Alternate Freeze Games with a Deep Reinforcement Learning Algorithm

Air-Combat Strategy Using Deep Q-Learning

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning

A Deep Reinforcement Learning Based Intelligent Decision Method for UCAV Air Combat

The Optimal Strategies of Maneuver Decision in Air Combat of UCAV Based on the Improved TD3 Algorithm

Curriculum-RL Based Air Combat Decision-Making

Hierarchical Reinforcement Learning from Competitive Self-play for Dual-aircraft formation air combat

A Decision-Making Method for Air Combat Maneuver Based on Hybrid Deep Learning Network

Deep Reinforcement Learning With Application to Air Confrontation Intelligent Decision-Making of Manned/Unmanned Aerial Vehicle Cooperative System

Deep Reinforcement-Learning-Based Air-Combat-Maneuver Generation Framework

Over-the-Horizon Air Combat Environment Modeling and Deep Reinforcement Learning Application

Air Combat Maneuver Decision Based on Deep Reinforcement Learning with Auxiliary Reward

Multi-intent autonomous decision-making for air combat with deep reinforcement learning