Abstract:Sharing a pre-trained machine learning model, particularly a deep neural network via prediction APIs, is becoming a common practice on machine learning as a service (MLaaS) platforms nowadays. Although deep neural networks (DNN) have shown remarkable successes in many tasks, they are also criticized for the lack of interpretability and transparency. Interpreting a shared DNN model faces two additional challenges compared with interpreting a general model. (1) Limited training data can be disclosed to users. (2) The internal structure of the models may not be available. These two challenges impede the application of most existing interpretability approaches, such as saliency maps or influence functions, for DNN models. Case-based reasoning methods have been used for interpreting decisions; however, how to select and organize the data points under the constraints of shared DNN models is not discussed. Moreover, simply providing cases as explanations may not be sufficient for supporting instance level interpretability. Meanwhile, existing interpretation methods for DNN models generally lack the means to evaluate the reliability of the interpretation. In this article, we propose a framework named Shared Model INTerpreter (SMINT) to address the above limitations. We propose a new data structure called a boundary graph to organize training points to mimic the predictions of DNN models. We integrate local features, such as saliency maps and interpretable input masks, into the data structure to help users to infer the model decision boundaries. We show that the boundary graph is able to address the reliability issues in many local interpretation methods. We further design an algorithm named hidden-layer aware p-test to measure the reliability of the interpretations. Our experiments show that SMINT is able to achieve above 99% fidelity to corresponding DNN models on both MNIST and ImageNet by sharing only a tiny fraction of training data to make these models interpretable. The human pilot study demonstrates that SMINT provides better interpretability compared with existing methods. Moreover, we demonstrate that SMINT is able to assist model tuning for better performance on different user data.

Interpretable Deep Convolutional Neural Networks via Meta-learning

Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

Hybrid CNN -Interpreter: Interpret local and global contexts for CNN-based Models

Deeper Interpretability of Deep Networks

Interpreting the decisions of CNNs via influence functions

Visualizing Deep Networks Using Segmentation Recognition and Interpretation Algorithm

Visual Interpretability forDeepLearning

Interpretability of deep learning models: A survey of results

High-precision Linearized Interpretation for Fully Connected Neural Network

Interpretable neural networks: principles and applications

Visual Interpretability for Deep Learning: a Survey

Interpretable Neural Network Decoupling.

SMINT

Interpreting Deep Learning Model Using Rule-based Method

Finding Representative Interpretations on Convolutional Neural Networks

A Survey of the Interpretability Aspect of Deep Learning Models

FICNN: A Framework for the Interpretation of Deep Convolutional Neural Networks

Interpretability for Reliable, Efficient, and Self-Cognitive DNNs: from Theories to Applications.

An Interpretable Neural Network Model Through Piecewise Linear Approximation

Interpreting Convolutional Neural Network by Joint Evaluation of Multiple Feature Maps and an Improved NSGA-II Algorithm

Interpretable Text Classification Using CNN and Max-pooling