Scaling Intelligent Agents in Combat Simulations for Wargaming

Scotty Black,Christian Darken

2024-02-09

Abstract:Remaining competitive in future conflicts with technologically-advanced competitors requires us to accelerate our research and development in artificial intelligence (AI) for wargaming. More importantly, leveraging machine learning for intelligent combat behavior development will be key to one day achieving superhuman performance in this domain--elevating the quality and accelerating the speed of our decisions in future wars. Although deep reinforcement learning (RL) continues to show promising results in intelligent agent behavior development in games, it has yet to perform at or above the human level in the long-horizon, complex tasks typically found in combat modeling and simulation. Capitalizing on the proven potential of RL and recent successes of hierarchical reinforcement learning (HRL), our research is investigating and extending the use of HRL to create intelligent agents capable of performing effectively in these large and complex simulation environments. Our ultimate goal is to develop an agent capable of superhuman performance that could then serve as an AI advisor to military planners and decision-makers. This papers covers our ongoing approach and the first three of our five research areas aimed at managing the exponential growth of computations that have thus far limited the use of AI in combat simulations: (1) developing an HRL training framework and agent architecture for combat units; (2) developing a multi-model framework for agent decision-making; (3) developing dimension-invariant observation abstractions of the state space to manage the exponential growth of computations; (4) developing an intrinsic rewards engine to enable long-term planning; and (5) implementing this framework into a higher-fidelity combat simulation.

Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the problem of developing artificial intelligence agents capable of making efficient decisions in war simulations to support military planning and decision-making. Specifically, the paper focuses on how to use Hierarchical Reinforcement Learning (HRL) to create intelligent agents that perform excellently in complex combat environments, achieving superhuman levels of performance. These issues include:

1. **Expanding the application of artificial intelligence in complex combat simulations**: Current reinforcement learning methods perform poorly when dealing with long-term, complex tasks, especially in combat modeling and simulation. The paper proposes to address this issue through hierarchical reinforcement learning to tackle the computational explosion problem in large-scale and complex scenarios.

2. **Improving decision quality and speed**: By developing high-performance artificial intelligence agents, the paper aims to support military planners and decision-makers, accelerating the decision-making process and enhancing decision quality. These agents can provide strategic advantages in future wars.

3. **Managing computational complexity**: The paper proposes various methods to manage and reduce computational complexity, including developing hierarchical reinforcement learning training frameworks, multi-model decision frameworks, dimension-invariant state space abstraction, intrinsic reward mechanisms, etc.

4. **Achieving superhuman-level intelligent agents**: The ultimate goal is to develop intelligent agents that can surpass human performance, serving as support tools for military planning and decision-making, helping to address future technological challenges and strategic threats.

In summary, the paper aims to address the challenges faced by current artificial intelligence in complex combat simulations through hierarchical reinforcement learning and other technical means, providing stronger support for military decision-making.