Abstract:The field of transparent Machine Learning (ML) has contributed many novel methods aiming at better interpretability for computer vision and ML models in general. But how useful the explanations provided by transparent ML methods are for humans remains difficult to assess. Most studies evaluate interpretability in qualitative comparisons, they use experimental paradigms that do not allow for direct comparisons amongst methods or they report only offline experiments with no humans in the loop. While there are clear advantages of evaluations with no humans in the loop, such as scalability, reproducibility and less algorithmic bias than with humans in the loop, these metrics are limited in their usefulness if we do not understand how they relate to other metrics that take human cognition into account. Here we investigate the quality of interpretable computer vision algorithms using techniques from psychophysics. In crowdsourced annotation tasks we study the impact of different interpretability approaches on annotation accuracy and task time. In order to relate these findings to quality measures for interpretability without humans in the loop we compare quality metrics with and without humans in the loop. Our results demonstrate that psychophysical experiments allow for robust quality assessment of transparency in machine learning. Interestingly the quality metrics computed without humans in the loop did not provide a consistent ranking of interpretability methods nor were they representative for how useful an explanation was for humans. These findings highlight the potential of methods from classical psychophysics for modern machine learning applications. We hope that our results provide convincing arguments for evaluating interpretability in its natural habitat, human-ML interaction, if the goal is to obtain an authentic assessment of interpretability.

Evaluation of post-hoc interpretability methods in time-series classification

InterpretTime: a new approach for the systematic evaluation of neural-network interpretability in time series classification

Revisiting the robustness of post-hoc interpretability methods

A psychophysics approach for quantitative comparison of interpretable computer vision models

Benchmarking Counterfactual Interpretability in Deep Learning Models for Time Series Classification

Interpretation of Time-Series Deep Models: A Survey

Machine Learning Interpretability: A Survey on Methods and Metrics

Post-hoc Interpretability for Neural NLP: A Survey

Towards a Unified Framework for Evaluating Explanations

Issues with post-hoc counterfactual explanations: a discussion

Benchmarking Deep Learning Interpretability in Time Series Predictions

Improving the Evaluation and Actionability of Explanation Methods for Multivariate Time Series Classification

InterpretCC: Intrinsic User-Centric Interpretability through Global Mixture of Experts

The Logic Traps in Evaluating Post-hoc Interpretations.

A Survey of the Interpretability Aspect of Deep Learning Models

Quantifying Interpretability and Trust in Machine Learning Systems

Interpretability of Machine Learning Methods Applied to Neuroimaging

An Interpretable Probabilistic Approach for Demystifying Black-box Predictive Models

An Empirical Comparison of Interpretable Models to Post-Hoc Explanations

Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods