Abstract:As Unmanned Combat Aerial Vehicle (UCAV) continue to play an increasingly pivotal role in modern aerial warfare, enhancing their intelligence levels is imperative for global military advancement. Despite notable progress in employing deep reinforcement learning for autonomous air combat maneuver decision-making, existing methods grapple with subpar performance, sluggish training, and susceptibility to local optima. Therefore, this paper proposes a new air combat maneuver decision algorithm based on Proximal Policy Optimization (PPO). Firstly, we establish a UCAV adversarial model and design a dual observation space. Secondly, we develop an Actor-Critic network based on Bidirectional Long Short-Term Memory (BiLSTM) and Multi-Head Self-Attention (MHSA), which better handles high-dimensional information with temporal correlations in air combat situations. Thirdly, we propose an action selection method based on Parallel Monte Carlo Tree Search with Watch the Unobserved (WU-PMCTS) to assist the algorithm in making more effective maneuver decisions. Fourthly, we design a Dynamic Reward Evaluation (DRE) method to dynamically adjust the weights of various rewards according to different adversarial situations, improving algorithm performance. Finally, we introduce an Advantage Prioritized Experience Replay (APER) to sample according to the sample advantage values, enhancing algorithm training efficiency. Experimental results from ablation and comparative experiments demonstrate the superiority of the proposed algorithm over PPO and other mainstream algorithms, with a 0.32 increase in average return and a 36% increase in win rate.

UCAV Air Combat Maneuver Decisions Based on a Proximal Policy Optimization Algorithm with Situation Reward Shaping

Autonomous Air Combat Maneuver Decision-Making Based on PPO-BWDA

UCAV Autonomous Maneuvering Decision Based on Curriculum Learning Mechanism Training

Mean policy-based proximal policy optimization for maneuvering decision in multi-UAV air combat

Manoeuvre Decision‐making of Unmanned Aerial Vehicles in Air Combat Based on an Expert Actor‐based Soft Actor Critic Algorithm

Research on Autonomous Maneuvering Decision of UCAV based on Approximate Dynamic Programming

UAV maneuver decision-making via deep reinforcement learning for short-range air combat

Autonomous Maneuver Decision for Unmanned Aerial Vehicle Via Improved Pigeon-Inspired Optimization

Maneuver Strategy Generation of UCAV for Within Visual Range Air Combat Based on Multi-Agent Reinforcement Learning and Target Position Prediction

The Optimal Strategies of Maneuver Decision in Air Combat of UCAV Based on the Improved TD3 Algorithm

UAV Cooperative Air Combat Maneuvering Decision-Making Using GRU-MAPPO

Maneuver and Attack Strategy Generation Method for Autonomous Air Combat in Hybrid Action Space Based on Proximal Policy Optimization

Autonomous Maneuver Decision of UCAV Air Combat Based on Double Deep Q Network Algorithm and Stochastic Game Theory

Autonomous Maneuver Decisions Via Transfer Learning Pigeon-Inspired Optimization for UCAVs in Dogfight Engagements

One-to-one Close Air Combat Maneuver Decision Method Based on Target Maneuver Intention Prediction

Research on UCAV Maneuvering Decision Method Based on Heuristic Reinforcement Learning

Cooperative Decision-Making Algorithm with Efficient Convergence for UCAV Formation in Beyond-Visual-range Air Combat Based on Multi-Agent Reinforcement Learning

UAV Cooperative Air Combat Maneuvering Confrontation Based on Multi-agent Reinforcement Learning

Hierarchical Maneuver Decision Method Based on PG-Option for UAV Pursuit-Evasion Game

Air Combat Autonomous Maneuver Decision for One-on-one Within Visual Range Engagement Base on Robust Multi-Agent Reinforcement Learning.

Multi UCAVs Coordinated Attack Target Assignment Using an Improved PSO Algorithm