Abstract:Distributed decision-making in multi-agent systems presents difficult challenges for interactive behavior learning in both cooperative and competitive systems. To mitigate this complexity, MAIDRL presents a semi-centralized Dense Reinforcement Learning algorithm enhanced by agent influence maps (AIMs), for learning effective multi-agent control on StarCraft Multi-Agent Challenge (SMAC) scenarios. In this paper, we extend the DenseNet in MAIDRL and introduce semi-centralized Multi-Agent Dense-CNN Reinforcement Learning, MAIDCRL, by incorporating convolutional layers into the deep model architecture, and evaluate the performance on both homogeneous and heterogeneous scenarios. The results show that the CNN-enabled MAIDCRL significantly improved the learning performance and achieved a faster learning rate compared to the existing MAIDRL, especially on more complicated heterogeneous SMAC scenarios. We further investigate the stability and robustness of our model. The statistics reflect that our model not only achieves higher winning rate in all the given scenarios but also boosts the agent's learning process in fine-grained decision-making.

What problem does this paper attempt to address?

This paper is primarily dedicated to addressing the distributed decision-making problem in Multi-Agent Systems (MAS), particularly within the game environment of StarCraft II (SC2). Specifically, the paper attempts to solve the following key issues: 1. **Extending traditional single-agent Reinforcement Learning (RL) methods to adapt to multi-agent environments**: Traditional RL algorithms face challenges when dealing with multi-agent systems, such as the state space growing exponentially with the number of agents, and learning stability issues caused by the Non-Markovian Property. 2. **Proposing a new semi-centralized multi-agent intensive convolutional neural network reinforcement learning algorithm (MAIDCRL)**: This algorithm integrates Agent Influence Maps (AIMs) and Convolutional Neural Networks (CNNs) to achieve effective and efficient multi-agent control in the StarCraft Multi-Agent Challenge (SMAC) scenarios. 3. **Improving learning performance**: By incorporating CNNs into the original Multi-Agent Influence Dense Reinforcement Learning (MAIDRL) framework, MAIDCRL demonstrates significantly improved learning performance in various homogeneous and heterogeneous SMAC scenarios, including faster learning speed and higher win rates. 4. **Enhancing the robustness and stability of the algorithm**: Through the analysis of results from multiple rounds of experiments in different scenarios, it is verified that MAIDCRL has advantages over MAIDRL in terms of learning stability and robustness. 5. **Behavior analysis**: By qualitatively analyzing the learned behaviors, the study reveals effective strategies adopted by agents during the game, such as prioritizing cooperative attacks and minimal movement adjustments for positioning. In summary, this research aims to address the challenges faced by reinforcement learning in multi-agent environments by proposing a new multi-agent reinforcement learning algorithm—MAIDCRL, and demonstrates its superiority in the complex game of StarCraft II.

MAIDCRL: Semi-centralized Multi-Agent Influence Dense-CNN Reinforcement Learning

S2rl

Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?

SC-MAIRL: Semi-Centralized Multi-Agent Imitation Reinforcement Learning

Consciousness-Aware Multi-Agent Reinforcement Learning

DMADRL: A Distributed Multi-agent Deep Reinforcement Learning Algorithm for Cognitive Offloading in Dynamic MEC Networks

Coordination as inference in multi-agent reinforcement learning

Learning Controlled and Targeted Communication with the Centralized Critic for the Multi-Agent System.

Enhancing cooperation by cognition differences and consistent representation in multi-agent reinforcement learning

Learning Attentional Communication with a Common Network for Multiagent Reinforcement Learning.

MARL-LNS: Cooperative Multi-agent Reinforcement Learning via Large Neighborhoods Search

Hybrid Centralized Training and Decentralized Execution Reinforcement Learning in Multi-Agent Path-Finding Simulations

Cooperative multi-agent game based on reinforcement learning

Adaptive Traffic Signal Control for Large-Scale Scenario with Cooperative Group-based Multi-agent Reinforcement Learning

Enabling Multi-Agent Transfer Reinforcement Learning via Scenario Independent Representation

Attentive Relational State Representation in Decentralized Multiagent Reinforcement Learning.

MANSA: Learning Fast and Slow in Multi-Agent Systems

Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Control

Efficient and scalable reinforcement learning for large-scale network control

Effective Master-Slave Communication On A Multi-Agent Deep Reinforcement Learning System

CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learning