Abstract:While Explainable Artificial Intelligence (XAI) is increasingly expanding more areas of application, little has been applied to make deep Reinforcement Learning (RL) more comprehensible. As RL becomes ubiquitous and used in critical and general public applications, it is essential to develop methods that make it better understood and more interpretable. This study proposes a novel approach to explain cooperative strategies in multiagent RL using Shapley values, a game theory concept used in XAI that successfully explains the rationale behind decisions taken by Machine Learning algorithms. Through testing common assumptions of this technique in two cooperation-centered socially challenging multi-agent environments environments, this article argues that Shapley values are a pertinent way to evaluate the contribution of players in a cooperative multi-agent RL context. To palliate the high overhead of this method, Shapley values are approximated using Monte Carlo sampling. Experimental results on Multiagent Particle and Sequential Social Dilemmas show that Shapley values succeed at estimating the contribution of each agent. These results could have implications that go beyond games in economics, (e.g., for non-discriminatory decision making, ethical and responsible AI-derived decisions or policy making under fairness constraints). They also expose how Shapley values only give general explanations about a model and cannot explain a single run, episode nor justify precise actions taken by agents. Future work should focus on addressing these critical aspects.

Coalitional Strategies for Efficient Individual Prediction Explanation

Approximation of group explainers with coalition structure using Monte Carlo sampling on the product space of coalitions and features

Shapley Values for Explaining the Black Box Nature of Machine Learning Model Clustering

A novel approach to explain the black-box nature of machine learning in compressive strength predictions of concrete using Shapley additive explanations (SHAP)

On the Relationship Between Explanation and Prediction: A Causal View

A Unified Approach to Interpreting Model Predictions

Conversations that explain predictions

Succint Interaction-Aware Explanations

Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values

SynthTree: Co-supervised Local Model Synthesis for Explainable Prediction

Understanding Model Predictions: A Comparative Analysis of SHAP and LIME on Various ML Algorithms

Do intermediate feature coalitions aid explainability of black-box models?

Explaining machine learning models using entropic variable projection

Shaping Up SHAP: Enhancing Stability through Layer-Wise Neighbor Selection

Prediction can be safely used as a proxy for explanation in causally consistent Bayesian generalized linear models

Interpretable machine learning with tree-based shapley additive explanations: Application to metabolomics datasets for binary classification

Local Interpretable Model Agnostic Shap Explanations for machine learning models

Considerations When Learning Additive Explanations for Black-Box Models

Selective Explanations

CohEx: A Generalized Framework for Cohort Explanation

Constraint-Driven Explanations for Black-Box ML Models