Abstract:While achieving tremendous success in various fields, existing multi-agentreinforcement learning (MARL) with a black-box neural network architecturemakes decisions in an opaque manner that hinders humans from understanding thelearned knowledge and how input observations influence decisions. Instead,existing interpretable approaches, such as traditional linear models anddecision trees, usually suffer from weak expressivity and low accuracy. Toaddress this apparent dichotomy between performance and interpretability, oursolution, MIXing Recurrent soft decision Trees (MIXRTs), is a novelinterpretable architecture that can represent explicit decision processes viathe root-to-leaf path and reflect each agent's contribution to the team.Specifically, we construct a novel soft decision tree to address partialobservability by leveraging the advances in recurrent neural networks, anddemonstrate which features influence the decision-making process through thetree-based model. Then, based on the value decomposition framework, we linearlyassign credit to each agent by explicitly mixing individual action values toestimate the joint action value using only local observations, providing newinsights into how agents cooperate to accomplish the task. Theoretical analysisshows that MIXRTs guarantees the structural constraint on additivity andmonotonicity in the factorization of joint action values. Evaluations on thechallenging Spread and StarCraft II tasks show that MIXRTs achieves competitiveperformance compared to widely investigated methods and delivers morestraightforward explanations of the decision processes. We explore a promisingpath toward developing learning algorithms with both high performance andinterpretability, potentially shedding light on new interpretable paradigms forMARL.

Interpretable Reinforcement Learning with Ensemble Methods

Inherently Interpretable Tree Ensemble Learning

Ensemble Interpretation: A Unified Method for Interpretable Machine Learning

Deep interpretable ensembles

A survey on interpretable reinforcement learning

Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy

A Unified Approach to Extract Interpretable Rules from Tree Ensembles via Integer Programming

Social Interpretable Reinforcement Learning

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

MIXRTs: Toward Interpretable Multi-Agent Reinforcement Learning Via Mixing Recurrent Soft Decision Trees

Better Short than Greedy: Interpretable Models through Optimal Rule Boosting

A stable data-augmented reinforcement learning method with ensemble exploration and exploitation

Unboxing Tree Ensembles for interpretability: a hierarchical visualization tool and a multivariate optimal re-built tree

Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning

Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression

Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning

Optimal Interpretability-Performance Trade-off of Classification Trees with Black-Box Reinforcement Learning

BET: Explaining Deep Reinforcement Learning through The Error-Prone Decisions

Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning

Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis

Learning Two-Step Hybrid Policy for Graph-Based Interpretable Reinforcement Learning