Abstract:In today's rapidly evolving military landscape, advancing artificial intelligence (AI) in support of wargaming becomes essential. Despite reinforcement learning (RL) showing promise for developing intelligent agents, conventional RL faces limitations in handling the complexity inherent in combat simulations. This dissertation proposes a comprehensive approach, including targeted observation abstractions, multi-model integration, a hybrid AI framework, and an overarching hierarchical reinforcement learning (HRL) framework. Our localized observation abstraction using piecewise linear spatial decay simplifies the RL problem, enhancing computational efficiency and demonstrating superior efficacy over traditional global observation methods. Our multi-model framework combines various AI methodologies, optimizing performance while still enabling the use of diverse, specialized individual behavior models. Our hybrid AI framework synergizes RL with scripted agents, leveraging RL for high-level decisions and scripted agents for lower-level tasks, enhancing adaptability, reliability, and performance. Our HRL architecture and training framework decomposes complex problems into manageable subproblems, aligning with military decision-making structures. Although initial tests did not show improved performance, insights were gained to improve future iterations. This study underscores AI's potential to revolutionize wargaming, emphasizing the need for continued research in this domain.

What problem does this paper attempt to address?

The main problem this paper attempts to address is how to develop more intelligent combat simulation agents through Hierarchical Reinforcement Learning (HRL) in modern warfare simulations. Although existing Artificial Intelligence (AI) systems have achieved some success in simpler scenarios, their performance often falls short in more complex warfare simulation environments. To tackle this challenge, the paper proposes improvements to current AI and Reinforcement Learning (RL) methods, including: 1. **Novel Observation Abstraction**: Develop a local observation abstraction method that simplifies the RL problem using piecewise linear spatial decay, improving computational efficiency and outperforming traditional global observation methods. 2. **Multi-Model Approach**: Combine multiple AI methods to optimize performance while allowing the use of diverse, specialized behavior models. 3. **Hybrid AI Systems**: Integrate RL with scripted agents, using RL for high-level decision-making and scripted agents for low-level tasks, enhancing adaptability, reliability, and performance. 4. **Hierarchical Reinforcement Learning Framework**: Construct an HRL architecture and training framework that decomposes complex problems into manageable sub-problems, aligning with military decision-making structures. These improvements aim to enhance the scalability of AI and the effectiveness of agents, particularly in combat simulation environments of increasing complexity. The short-term goal of the paper is to improve the intelligence and adaptability of agents, ultimately providing valuable insights and tools for wargame applications to support education, training, analysis, operational planning, and decision-making. The long-term vision is to leverage AI capabilities to improve the quality and speed of decision-making in the complex environments of modern battlefields.

Mastering the Digital Art of War: Developing Intelligent Combat Simulation Agents for Wargaming Using Hierarchical Reinforcement Learning

Scaling Intelligent Agents in Combat Simulations for Wargaming

Scaling intelligent agent combat behaviors through hierarchical reinforcement learning

Scaling Artificial Intelligence for Digital Wargaming in Support of Decision-Making

Localized Observation Abstraction Using Piecewise Linear Spatial Decay for Reinforcement Learning in Combat Simulations

Large Scale Deep Reinforcement Learning in War-games.

Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering

Intelligent Decision-Making and Human Language Communication Based on Deep Reinforcement Learning in a Wargame Environment

Curriculum-RL Based Air Combat Decision-Making

Hierarchical Reinforcement Learning from Competitive Self-play for Dual-aircraft formation air combat

Effective Master-Slave Communication On A Multi-Agent Deep Reinforcement Learning System

Hierarchical Reinforcement Learning for Air-to-Air Combat

Intuitionistic Fuzzy MADM in Wargame Leveraging with Deep Reinforcement Learning

Enhancing multi-UAV air combat decision making via hierarchical reinforcement learning

Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning

Coordinating Multi-Agent Deep Reinforcement Learning in Wargame

Multi-agent Cooperative Combat Simulation in Naval Battlefield with Reinforcement Learning

Strategic maneuver and disruption with reinforcement learning approaches for multi-agent coordination

Self-play Decision-making Method of Deep Reinforcement Learning Guided by Behavior Tree under Complex Environment

Mastering air combat game with deep reinforcement learning

System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games