Abstract:Artificial neural networks (ANNs) are powerful tools for data analysis and are particularly suitable for modeling relationships between variables for best prediction of an outcome. While these models can be used to answer many important research questions, their utility has been critically limited because the interpretation of the "black box" model is difficult. Clinical investigators usually employ ANN models to predict the clinical outcomes or to make a diagnosis; the model however is difficult to interpret for clinicians. To address this important shortcoming of neural network modeling methods, we describe several methods to help subject-matter audiences (e.g., clinicians, medical policy makers) understand neural network models. Garson's algorithm describes the relative magnitude of the importance of a descriptor (predictor) in its connection with outcome variables by dissecting the model weights. The Lek's profile method explores the relationship of the outcome variable and a predictor of interest, while holding other predictors at constant values (e.g., minimum, 20th quartile, maximum). While Lek's profile was developed specifically for neural networks, partial dependence plot is a more generic version that visualize the relationship between an outcome and one or two predictors. Finally, the local interpretable model-agnostic explanations (LIME) method can show the predictions of any classification or regression, by approximating it locally with an interpretable model. R code for the implementations of these methods is shown by using example data fitted with a standard, feed-forward neural network model. We offer codes and step-by-step description on how to use these tools to facilitate better understanding of ANN.

Interpreting Neural Networks through Mahalanobis Distance

Visualizing and Understanding Neural Models in NLP

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Understanding Neural Networks through Representation Erasure.

Opening the Black Box of Neural Networks: Methods for Interpreting Neural Network Models in Clinical Applications

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

Interpreting Deep Neural Networks Through Variable Importance

A Survey of the Interpretability Aspect of Deep Learning Models

An Interpretable Neural Network Model Through Piecewise Linear Approximation

How to Explain Neural Networks: an Approximation Perspective

Model-Agnostic Interpretation Framework in Machine Learning: A Comparative Study in NBA Sports

Closed-Form Interpretation of Neural Network Latent Spaces with Symbolic Gradients

Representations as Language: An Information-Theoretic Framework for Interpretability

Interpretability of deep learning models: A survey of results

Neural Networks Use Distance Metrics

A Survey on Neural Network Interpretability

Neural network interpretability with layer-wise relevance propagation: novel techniques for neuron selection and visualization

Interpreting Neural Networks With Nearest Neighbors

Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

Interpretable Function Embedding and Module in Convolutional Neural Networks

Interpretable Deep Convolutional Neural Networks via Meta-learning