Abstract:Many real-world cooperative problems can be implemented using Multi-Agent Reinforcement Learning (MARL) techniques, such as urban traffic control or multi-role games. However, the policy learning of MARL algorithms contains features of long-trajectory training and partial observability, which leads to the sparsity of reward and the lack of decision information. To solve the above issues, this article studies hierarchical deep MARL and proposes a novel model named Hierarchical Spatio-Temporal Communication Network (HSTCN). HSTCN designs hierarchical policies with two-time granularities: high-level and low-level policies. All agents are jointly entered into a joint policy containing the above two policies, and each has its execution policy. Specifically, the high-level policy provides intrinsic goals and continuous reward samples for the low-level policy to alleviate reward sparsity. The Low-level policy absorbs the above information to improve the efficiency of the agents' execution policies and interact with the environment to optimize the next reward. What's more, the high-level policy designs a graph-like structural model with Spatio-Temporal abstract. The Spatio-Temporal model expands receptive fields to receive neighborhood information and facilitates learning more robust policies by capturing the underlying graph's spatial dependencies and temporal dynamics. Meanwhile, an evaluation network is added to increase the robustness. Empirically, we demonstrated the effectiveness of HSTCN in a long-trajectory training environment through Simulation of Urban MObility (SUMO), while StarCraft II maps are tested as abstract environment. The above experimental results prove that the performance of HSTCN is superior to other advanced algorithms and verify the rationality of HSTCN design.

STAS: Spatial-Temporal Return Decomposition for Multi-agent Reinforcement Learning.

STAS: Spatial-Temporal Return Decomposition for Solving Sparse Rewards Problems in Multi-agent Reinforcement Learning

Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games

Expert demonstrations guide reward decomposition for multi-agent cooperation

Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning

Priority over Quantity: A Self-Incentive Credit Assignment Scheme for Cooperative Multiagent Reinforcement Learning

Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?

A Graph-Based Soft Actor Critic Approach in Multi-Agent Reinforcement Learning

Credit Assignment with Meta-Policy Gradient for Multi-Agent Reinforcement Learning

CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learning

Agent-Temporal Credit Assignment for Optimal Policy Preservation in Sparse Multi-Agent Reinforcement Learning

Credit assignment in heterogeneous multi-agent reinforcement learning for fully cooperative tasks

Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

Q-value Path Decomposition for Deep Multiagent Reinforcement Learning

AVD-Net: Attention Value Decomposition Network for Deep Multi-Agent Reinforcement Learning

Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

Multi-Level Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

Credit assignment with predictive contribution measurement in multi-agent reinforcement learning

Hierarchical Coordination Multi-Agent Reinforcement Learning with Spatio-Temporal Abstraction

Multiagent Q-learning with Sub-Team Coordination.