Abstract:Interpretability for neural networks is a trade-off between three key requirements: 1) faithfulness of the explanation (i.e., how perfectly it explains the prediction), 2) understandability of the explanation by humans, and 3) model performance. Most existing methods compromise one or more of these requirements; e.g., post-hoc approaches provide limited faithfulness, automatically identified feature masks compromise understandability, and intrinsically interpretable methods such as decision trees limit model performance. These shortcomings are unacceptable for sensitive applications such as education and healthcare, which require trustworthy explanations, actionable interpretations, and accurate predictions. In this work, we present InterpretCC (interpretable conditional computation), a family of interpretable-by-design neural networks that guarantee human-centric interpretability, while maintaining comparable performance to state-of-the-art models by adaptively and sparsely activating features before prediction. We extend this idea into an interpretable, global mixture-of-experts (MoE) model that allows humans to specify topics of interest, discretely separates the feature space for each data point into topical subnetworks, and adaptively and sparsely activates these topical subnetworks for prediction. We apply variations of the InterpretCC architecture for text, time series and tabular data across several real-world benchmarks, demonstrating comparable performance with non-interpretable baselines, outperforming interpretable-by-design baselines, and showing higher actionability and usefulness according to a user study.

InterpretTime: a new approach for the systematic evaluation of neural-network interpretability in time series classification

Evaluation of post-hoc interpretability methods in time-series classification

Interpretation of Time-Series Deep Models: A Survey

InterpretCC: Intrinsic User-Centric Interpretability through Global Mixture of Experts

Improving the Evaluation and Actionability of Explanation Methods for Multivariate Time Series Classification

Benchmarking Deep Learning Interpretability in Time Series Predictions

A Survey of the Interpretability Aspect of Deep Learning Models

Inherently Interpretable Time Series Classification via Multiple Instance Learning

Extracting Interpretable Local and Global Representations from Attention on Time Series

Explaining the Unexplained: Revealing Hidden Correlations for Better Interpretability

Explanation Space: A New Perspective into Time Series Interpretability

Time Interpret: a Unified Model Interpretability Library for Time Series

Using SHAP to Measure Interpretability of Neuronal Feature Visualization.

ST-Tree with Interpretability for Multivariate Time Series Classification

How good Neural Networks interpretation methods really are? A quantitative benchmark

A Survey on Neural Network Interpretability

Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

A Unified Approach to Interpreting Model Predictions

Towards Understanding Sensitive and Decisive Patterns in Explainable AI: A Case Study of Model Interpretation in Geometric Deep Learning

Interpretability of deep learning models: A survey of results

Machine Learning Interpretability: A Survey on Methods and Metrics