Abstract:Autonomous decision-making in unmanned aerial vehicle (UVA) confrontations presents challenges in making optimal strategy. Therefore, deep reinforcement learning (DRL) has been adopted to address these issues. However, existing DRL decision-making models suffer from poor situational awareness and inability to distinguish between different intentions. Therefore, a multi-intent autonomous decision-making is proposed in this paper. First, three typical intentions are designed comprising head-on attacking, pursuing and fleeing to derive decision models representing different intentions. Reinforcement learning based air combat game model is constructed with different intentions, which contains designing reward functions for intentions to deal with the problem of sparse rewards. Then, we propose the Temporal Proximal Policy Optimization (T-PPO) algorithm, which optimizes the Proximal Policy Optimization algorithm by integrating the long short-term memory network and feedforward neural network. This algorithm extracts the historical temporal information to enhance situational awareness. In addition, a basic-confrontation progressive training method is proposed to provide intention guidance and increase training diversity, which can improve learning efficiency and intelligent decision-making capability. Finally, experiments in our constructed UAV confrontation environment demonstrate that the proposed intentional decision models exhibit good performance in stability and learning efficiency, achieving high rewards, win rates, and low steps. Specifically, our autonomous decision-making increases win rate by 26% when head-on attacking and learning efficiency by 50% when pursuing. It is further proof of the potential and value of our multi-intent autonomous decision-making applications.

Hierarchical Reinforcement Learning from Competitive Self-play for Dual-aircraft formation air combat

Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering

Enhancing multi-UAV air combat decision making via hierarchical reinforcement learning

Hierarchical Reinforcement Learning for Air-to-Air Combat

Model-free Maneuvering Control of Fixed-Wing UAVs Based on Deep Reinforcement Learning

Hierarchical Decision and Control for Continuous Multitarget Problem: Policy Evaluation with Action Delay

Autonomous Maneuver Decision Making of Dual-UAV Cooperative Air Combat Based on Deep Reinforcement Learning

Deep Reinforcement-Learning-Based Air-Combat-Maneuver Generation Framework

Cross coordination of behavior clone and reinforcement learning for autonomous within-visual-range air combat

Air Combat Maneuver Decision Based on Deep Reinforcement Learning and Game Theory

Deep Reinforcement Learning With Application to Air Confrontation Intelligent Decision-Making of Manned/Unmanned Aerial Vehicle Cooperative System

UAV Cooperative Air Combat Maneuvering Confrontation Based on Multi-agent Reinforcement Learning

UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning

Intelligent air defense task assignment based on hierarchical reinforcement learning

A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat

Homotopy Based Reinforcement Learning with Maximum Entropy for Autonomous Air Combat

H3E: Learning air combat with a three-level hierarchical framework embedding expert knowledge

Deep reinforcement learning-based air combat maneuver decision-making: literature review, implementation tutorial and future direction

Maneuver Decision-Making For Autonomous Air Combat Through Curriculum Learning And Reinforcement Learning With Sparse Rewards

Multi-intent autonomous decision-making for air combat with deep reinforcement learning

An evolutionary multi-agent reinforcement learning algorithm for multi-UAV air combat