Abstract:Autonomous maneuvering decision-making is a crucial technology for Unmanned Aerial Vehicles (UAVs) to take the air domination in modern unmanned warfare. With the advantage of balancing exploration and exploitation, as well as the immediacy of end-to-end output by combining with deep neural network, multi-agent reinforcement learning (MARL) has made remarkable achievements in multi-UAV autonomous air combat maneuvering decision-making (MUAAMD). However, the implementation of effective cooperative policy learning remains a challenging issue for MARL methods with centralized training decentralized execution (CTDE) paradigm. This paper proposes a MARL-based method to improve the performance of cooperation in MUAAMD. Firstly, considering the constraints of dynamic and limited perception for UAVs in the realistic air combat scenario, the MUAAMD problem is formulated based on partially observable Markov game (POMG) model. Secondly, a novel efficient MARL algorithm named the mean policy-based proximal policy optimization (MP3O) is introduced. Specifically, a joint policy optimization mechanism is constructed by estimating the policies of neighboring agents in group as a mean-field approximation while training, which enables both centralized evaluation and improvement of cooperative policy under the CTDE paradigm. Thirdly, by combining with three improvement techniques, a cooperative decision-making framework for MUAAMD based on MP3O is proposed. Empirically, results of simulations and comparative experiments validate the effectiveness of proposed method in promoting cooperative policy learning in resolving MUAAMD problem.

An Off-COMA Algorithm for Multi-UCAV Intelligent Combat Decision-Making

Cooperative Decision-Making Algorithm with Efficient Convergence for UCAV Formation in Beyond-Visual-range Air Combat Based on Multi-Agent Reinforcement Learning

Multi-UAV Cooperative Air Combat Decision-Making Based on Multi-Agent Double-Soft Actor-Critic

Air Combat Autonomous Maneuver Decision for One-on-one Within Visual Range Engagement Base on Robust Multi-Agent Reinforcement Learning.

UCAV Air Combat Maneuver Decisions Based on a Proximal Policy Optimization Algorithm with Situation Reward Shaping

Pso-Based Multi Ucavs Cooperative Attack Tasks Allocation And Its Simulation

Autonomous attack decision-making of UCAV based on ABFCM model framework

Multi UCAVs Coordinated Attack Target Assignment Using an Improved PSO Algorithm

Maneuver Strategy Generation of UCAV for Within Visual Range Air Combat Based on Multi-Agent Reinforcement Learning and Target Position Prediction

UAV Cooperative Air Combat Maneuvering Confrontation Based on Multi-agent Reinforcement Learning

Multi-UCAVs task assignment simulation platform based on parallel ant colony optimization

Research on UCAV Maneuvering Decision Method Based on Heuristic Reinforcement Learning

Autonomous Maneuver Decision of UCAV Air Combat Based on Double Deep Q Network Algorithm and Stochastic Game Theory

An evolutionary multi-agent reinforcement learning algorithm for multi-UAV air combat

Multi-UAV Counter-Game Model Based on Uncertain Information

UAV Swarm Air Combat Maneuver Decision-Making Method Based on Multi-Agent Reinforcement Learning and Transferring

Collaborative Decision-Making Method for Multi-UAV Based on Multiagent Reinforcement Learning

UAV Swarm Confrontation Using Hierarchical Multiagent Reinforcement Learning

Game of Drones: Intelligent Online Decision Making of Multi-UAV Confrontation

Multi-Uav Automatic Dynamic Obstacle Avoidance With Experience-Shared A2c

Mean policy-based proximal policy optimization for maneuvering decision in multi-UAV air combat