Abstract:Sharing a pre-trained machine learning model, particularly a deep neural network via prediction APIs, is becoming a common practice on machine learning as a service (MLaaS) platforms nowadays. Although deep neural networks (DNN) have shown remarkable successes in many tasks, they are also criticized for the lack of interpretability and transparency. Interpreting a shared DNN model faces two additional challenges compared with interpreting a general model. (1) Limited training data can be disclosed to users. (2) The internal structure of the models may not be available. These two challenges impede the application of most existing interpretability approaches, such as saliency maps or influence functions, for DNN models. Case-based reasoning methods have been used for interpreting decisions; however, how to select and organize the data points under the constraints of shared DNN models is not discussed. Moreover, simply providing cases as explanations may not be sufficient for supporting instance level interpretability. Meanwhile, existing interpretation methods for DNN models generally lack the means to evaluate the reliability of the interpretation. In this article, we propose a framework named Shared Model INTerpreter (SMINT) to address the above limitations. We propose a new data structure called a boundary graph to organize training points to mimic the predictions of DNN models. We integrate local features, such as saliency maps and interpretable input masks, into the data structure to help users to infer the model decision boundaries. We show that the boundary graph is able to address the reliability issues in many local interpretation methods. We further design an algorithm named hidden-layer aware p-test to measure the reliability of the interpretations. Our experiments show that SMINT is able to achieve above 99% fidelity to corresponding DNN models on both MNIST and ImageNet by sharing only a tiny fraction of training data to make these models interpretable. The human pilot study demonstrates that SMINT provides better interpretability compared with existing methods. Moreover, we demonstrate that SMINT is able to assist model tuning for better performance on different user data.

Interpretation of NLP models through input marginalization

Understanding Neural Networks through Representation Erasure.

Visualizing and Understanding Neural Models in NLP

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Interpreting Deep Learning Models in Natural Language Processing: A Review

Opening the Black Box of Neural Networks: Methods for Interpreting Neural Network Models in Clinical Applications

AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

On the Sensitivity and Stability of Model Interpretations in NLP

On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs

On the Lack of Robust Interpretability of Neural Text Classifiers

SMINT

Post-hoc Interpretability for Neural NLP: A Survey

Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions

How to Explain Neural Networks: an Approximation Perspective

Interpretation of Neural Networks Is Fragile

Knockout: A simple way to handle missing inputs

Fooling Neural Network Interpretations via Adversarial Model Manipulation

Using Interpretation Methods for Model Enhancement

Interpretability as Approximation: Understanding Black-Box Models by Decision Boundary

Interpreting Neural Networks With Nearest Neighbors