Abstract:With the rapid development of intelligent and informationized air battlefields,intelligent air combat has increasingly become key to affecting the outcome of a battlefield. In conventional multi-aircraft air combat,there are issues of low efficiency in intelligent decision-making,difficulty in meeting the needs of complex air combat environments,and unreasonable target allocation. In response to the problems in conventional multi-aircraft air combat,we introduce a long short-term memory-proximal policy optimization algorithm (LSTM-PPO). Using the long short-term memory network to extract features and perceive the situation of the state,an intelligent agent trains the normalized and feature-fused state information residual network and value network,chooses the optimal action through the proximal policy optimization strategy based on the current situation,and embeds a reward function containing expert knowledge during the training process to solve the problem of sparse rewards. Meanwhile,a target allocation algorithm based on threat value calculation is presented. Using angle,speed,and height threat values as the basis for target allocation,the ID of the target aircraft with the highest threat value on the battlefield is calculated in real-time. When the strategy network outputs an action of attack,it conducts target allocation. To confirm the effectiveness of the algorithm,we carried out 4v4 multi-aircraft air combat experiments in a digital twin simulation environment built by our research group. The red team consists of reinforcement learning agents based on LSTM-PPO algorithm,whereas the blue team comprises a finite state machine composed of expert knowledge bases. After more than 1200 rounds of aerial confrontation,the algorithm has been converged,and the win rate of the red team has reached 82％. Furthermore,we assessed the performance of four other mainstream reinforcement learning algorithms in 4v4 air combat experiments under the same experimental conditions. It is shown that the deep Q-network (DQN) and soft actor-critic (SAC) algorithms have difficulties in dealing with high-dimensional continuous action spaces and multiagent collaboration. The multi-agent deep deterministic policy gradient algorithm (MADDPG) employs a multi-agent strategy and cooperative training,so it exhibits a significantly higher win rate than the DQN and SAC algorithms. The multi-agent proximal policy optimization (MAPPO) algorithm has a relatively high failure rate and is not stable enough to deal with enemy aircraft's strategies in some cases. The LSTM-PPO algorithm shows a significantly higher win rate than other mainstream reinforcement learning algorithms in multi-aircraft collaborative air combat,which confirms the effectiveness of the LSTM-PPO algorithm in dealing with high-dimensional continuous action spaces and multi-aircraft collaborative operations.

Optimization of Stratospheric Airship Station-Keeping Strategy Based on LSTM-DQN Algorithm

Autonomous Trajectory Planning Method for Stratospheric Airship Regional Station-Keeping Based on Deep Reinforcement Learning

Spacecraft Attitude Maneuver Planning Based on Deep Reinforcement Learning under Complex Constraints

Model-free Maneuvering Control of Fixed-Wing UAVs Based on Deep Reinforcement Learning

Path planning of stratospheric airship in dynamic wind field based on deep reinforcement learning

Air-Combat Strategy Using Deep Q-Learning

Station-Keeping Control of Stratospheric Balloons Based on Simultaneous Optimistic Optimization in Dynamic Wind

Stratospheric wind field feature extraction and energy management for hybrid electric solar airship with deep reinforcement learning

Deep Q-Network Algorithm-Based Cyclic Air Braking Strategy for Heavy-Haul Trains

Real-Time Prediction of Multi-Degree-of-Freedom Ship Motion and Resting Periods Using LSTM Networks

LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle

OPTIMIZATION ALGORITHM FOR INTERPLANETARY TRANSFER TRAJECTORIES OF SOLAR SAILCRAFT BASED ON DEEP REINFORCEMENT LEARNING

Multi-constrained Intelligent Gliding Guidance Via Optimal Control and DQN

Harnessing LSTM for Nonlinear Ship Deck Motion Prediction in UAV Autonomous Landing amidst High Sea States

Improving Maneuver Strategy in Air Combat by Alternate Freeze Games with a Deep Reinforcement Learning Algorithm

Resource-Constrained Station-Keeping for Helium Balloons using Reinforcement Learning

Digital Twin Assisted Task Assignment in Multi-UAV Systems: A Deep Reinforcement Learning Approach

A Reinforcement Learning Handover Parameter Adaptation Method Based on LSTM-Aided Digital Twin for UDN

An Improved Deep Reinforcement Learning Method for Dispatch Optimization Strategy of Modern Power Systems

Dynamic joint optimization of power generation and voyage scheduling in ship power system based on deep reinforcement learning

Intelligent Decision Making and Target Assignment of Multi-Aircraft Air Combat Based on the LSTM-PPO Algorithm