Abstract:Sharing a pre-trained machine learning model, particularly a deep neural network via prediction APIs, is becoming a common practice on machine learning as a service (MLaaS) platforms nowadays. Although deep neural networks (DNN) have shown remarkable successes in many tasks, they are also criticized for the lack of interpretability and transparency. Interpreting a shared DNN model faces two additional challenges compared with interpreting a general model. (1) Limited training data can be disclosed to users. (2) The internal structure of the models may not be available. These two challenges impede the application of most existing interpretability approaches, such as saliency maps or influence functions, for DNN models. Case-based reasoning methods have been used for interpreting decisions; however, how to select and organize the data points under the constraints of shared DNN models is not discussed. Moreover, simply providing cases as explanations may not be sufficient for supporting instance level interpretability. Meanwhile, existing interpretation methods for DNN models generally lack the means to evaluate the reliability of the interpretation. In this article, we propose a framework named Shared Model INTerpreter (SMINT) to address the above limitations. We propose a new data structure called a boundary graph to organize training points to mimic the predictions of DNN models. We integrate local features, such as saliency maps and interpretable input masks, into the data structure to help users to infer the model decision boundaries. We show that the boundary graph is able to address the reliability issues in many local interpretation methods. We further design an algorithm named hidden-layer aware p-test to measure the reliability of the interpretations. Our experiments show that SMINT is able to achieve above 99% fidelity to corresponding DNN models on both MNIST and ImageNet by sharing only a tiny fraction of training data to make these models interpretable. The human pilot study demonstrates that SMINT provides better interpretability compared with existing methods. Moreover, we demonstrate that SMINT is able to assist model tuning for better performance on different user data.

How good Neural Networks interpretation methods really are? A quantitative benchmark

Fooling Neural Network Interpretations - Adversarial Noise to Attack Images.

Interpretability of Machine Learning Methods Applied to Neuroimaging

Benchmarking Deep Learning Interpretability in Time Series Predictions

A Survey of the Interpretability Aspect of Deep Learning Models

Interpreting Deep Neural Networks Through Variable Importance

Opening the Black Box of Neural Networks: Methods for Interpreting Neural Network Models in Clinical Applications

Evaluating Neuron Interpretation Methods of NLP Models

A Survey on Neural Network Interpretability

Saliency Suppressed, Semantics Surfaced: Visual Transformations in Neural Networks and the Brain

Identifying Interpretable Visual Features in Artificial and Biological Neural Systems

Interpretation of Neural Networks Is Fragile

Saliency Methods are Encoders: Analysing Logical Relations Towards Interpretation

Interpretability of deep learning models: A survey of results

Local vs distributed representations: What is the right basis for interpretability?

A review and benchmark of feature importance methods for neural networks

Neural network interpretability with layer-wise relevance propagation: novel techniques for neuron selection and visualization

SMINT

Better Understanding Differences in Attribution Methods via Systematic Evaluations

On the Lack of Robust Interpretability of Neural Text Classifiers

Increasing Interpretability of Neural Networks By Approximating Human Visual Saliency