Abstract:ABSTRACTCentralized Training with Decentralized Execution (CTDE) has been a popular paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and is widely used in many real applications. One of the major challenges in the training process is credit assignment, which aims to deduce the contributions of each agent according to the global rewards. Existing credit assignment methods focus on either decomposing the joint value function into individual value functions or measuring the impact of local observations and actions on the global value function. These approaches lack a thorough consideration of the complicated interactions among multiple agents, leading to an unsuitable assignment of credit and subsequently mediocre results on MARL. We propose Shapley Counterfactual Credit Assignment, a novel method for explicit credit assignment which accounts for the coalition of agents. Specifically, Shapley Value and its desired properties are leveraged in deep MARL to credit any combinations of agents, which grants us the capability to estimate the individual credit for each agent. Despite this capability, the main technical difficulty lies in the computational complexity of Shapley Value who grows factorially as the number of agents. We instead utilize an approximation method via Monte Carlo sampling, which reduces the sample complexity while maintaining its effectiveness. We evaluate our method on StarCraft II benchmarks across different scenarios. Our method outperforms existing cooperative MARL algorithms significantly and achieves the state-of-the-art, with especially large margins on tasks with more severe difficulties.

Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis

Towards Practical Credit Assignment for Deep Reinforcement Learning

Shapley Q-Value: A Local Reward Approach to Solve Global Reward Games

Towards Causal Credit Assignment

A Survey of Temporal Credit Assignment in Deep Reinforcement Learning

Credit assignment with predictive contribution measurement in multi-agent reinforcement learning

Credit Assignment During Movement Reinforcement Learning.

Hindsight Network Credit Assignment: Efficient Credit Assignment in Networks of Discrete Stochastic Units

Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning

Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learning

Deep Reinforcement Learning with Credit Assignment for Combinatorial Optimization

Selective Credit Assignment

Credit Assignment: Challenges and Opportunities in Developing Human-like AI Agents

Learning to assign credit in reinforcement learning by incorporating abstract relations

On Credit Assignment in Hierarchical Reinforcement Learning

Priority over Quantity: A Self-Incentive Credit Assignment Scheme for Cooperative Multiagent Reinforcement Learning

An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning

Credit assignment in heterogeneous multi-agent reinforcement learning for fully cooperative tasks

Model-based Credit Assignment for Model-free Deep Reinforcement Learning

Disentangling causal effects for hierarchical reinforcement learning

Multi-Level Credit Assignment for Cooperative Multi-Agent Reinforcement Learning