Abstract:Reinforcement learning (RL) is a powerful technique for training intelligent agents, but understanding why these agents make specific decisions can be quite challenging. This lack of transparency in RL models has been a long-standing problem, making it difficult for users to grasp the reasons behind an agent's behaviour. Various approaches have been explored to address this problem, with one promising avenue being reward decomposition (RD). RD is appealing as it sidesteps some of the concerns associated with other methods that attempt to rationalize an agent's behaviour in a post-hoc manner. RD works by exposing various facets of the rewards that contribute to the agent's objectives during training. However, RD alone has limitations as it primarily offers insights based on sub-rewards and does not delve into the intricate cause-and-effect relationships that occur within an RL agent's neural model. In this paper, we present an extension of RD that goes beyond sub-rewards to provide more informative explanations. Our approach is centred on a causal learning framework that leverages information-theoretic measures for explanation objectives that encourage three crucial properties of causal factors: causal sufficiency, sparseness, and orthogonality. These properties help us distill the cause-and-effect relationships between the agent's states and actions or rewards, allowing for a deeper understanding of its decision-making processes. Our framework is designed to generate local explanations and can be applied to a wide range of RL tasks with multiple reward channels. Through a series of experiments, we demonstrate that our approach offers more meaningful and insightful explanations for the agent's action selections.

What problem does this paper attempt to address?

This paper aims to address the issue of lack of transparency in the decision-making process of Reinforcement Learning (RL) models. Specifically, the authors propose a new method to enhance interpretability through Causal State Distillation. Traditional explanation methods, such as saliency-based explanations, often fail to adequately reflect the causal relationships behind agent behavior and have limitations when applied to RL. Therefore, this paper introduces a causal learning framework that utilizes information-theoretic measures to extract causal relationships between agent states and actions or rewards, thereby providing a deeper understanding of the agent's decision-making process. This framework is capable of generating local explanations and is applicable to various RL tasks with multiple reward channels. Through a series of experiments, it is demonstrated that this method can provide more meaningful and profound explanations for the agent's behavior choices.

Causal State Distillation for Explainable Reinforcement Learning

Explainable Reinforcement Learning via a Causal World Model

Causal explanation for reinforcement learning: quantifying state and temporal importance

A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges

Experiential Explanations for Reinforcement Learning

A Closer Look at Reward Decomposition for High-Level Robotic Explanations

Explainable Reinforcement Learning Through a Causal Lens

Explainable Reinforcement Learning via Model Transforms

Explanation of Reinforcement Learning Model in Dynamic Multi-Agent System

Explanation of Reinforcement Learning Model in Dynamic Multi-Agent System

Contrastive Explanations for Reinforcement Learning in terms of Expected Consequences

Counterfactual Explainer Framework for Deep Reinforcement Learning Models Using Policy Distillation

Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning

Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes

Explain the Explainer: Interpreting Model-Agnostic Counterfactual Explanations of a Deep Reinforcement Learning Agent

Learning explainable task-relevant state representation for model-free deep reinforcement learning

CDT: Cascading Decision Trees for Explainable Reinforcement Learning

Tell me why! Explanations support learning relational and causal structure

Effective Interpretable Policy Distillation via Critical Experience Point Identification

On Causally Disentangled State Representation Learning for Reinforcement Learning based Recommender Systems

Rethinking State Disentanglement in Causal Reinforcement Learning