Abstract:In many practical applications, deep neural networks have been typically deployed to operate as a black box predictor. Despite the high amount of work on interpretability and high demand on the reliability of these systems, they typically still have to include a human actor in the loop, to validate the decisions and handle unpredictable failures and unexpected corner cases. This is true in particular for failure-critical application domains, such as medical diagnosis. We present a novel approach to explain and support an interpretation of the decision-making process to a human expert operating a deep learning system based on Convolutional Neural Network (CNN). By modeling activation statistics on selected layers of a trained CNN via Gaussian Mixture Models (GMM), we develop a novel perceptual code in binary vector space that describes how the input sample is processed by the CNN. By measuring distances between pairs of samples in this perceptual encoding space, for any new input sample, we can now retrieve a set of most perceptually similar and dissimilar samples from an existing atlas of labeled samples, to support and clarify the decision made by the CNN model. Possible uses of this approach include for example Computer-Aided Diagnosis (CAD) systems working with medical imaging data, such as Magnetic Resonance Imaging (MRI) or Computed Tomography (CT) scans. We demonstrate the viability of our method in the domain of medical imaging for patient condition diagnosis, as the proposed decision explanation method via similar ground truth domain examples (e.g. from existing diagnosis archives) will be interpretable by the operating medical personnel. Our results indicate that our method is capable of detecting distinct prediction strategies that enable us to identify the most similar predictions from an existing atlas.

Explaining the Behavior of Neuron Activations in Deep Neural Networks

Which Neural Network Makes More Explainable Decisions? an Approach Towards Measuring Explainability

Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes

Normalized Activation Function: Toward Better Convergence

Understanding CNN Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoning

Explaining Predictions of Deep Neural Classifier via Activation Analysis

Intriguing properties of neural networks

Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models

Under the Hood of Neural Networks: Characterizing Learned Representations by Functional Neuron Populations and Network Ablations

Deeper Interpretability of Deep Networks

Understanding the Role of Pathways in a Deep Neural Network

Neural Activation Patterns (NAPs): Visual Explainability of Learned Concepts

Understanding Neural Network Through Neuron Level Visualization.

Visualizing Deep Neural Networks with Topographic Activation Maps

What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space

Neural network interpretability with layer-wise relevance propagation: novel techniques for neuron selection and visualization

Neural networks behave as hash encoders: An empirical study

On Functional Activations in Deep Neural Networks

Neuro-Inspired Deep Neural Networks with Sparse, Strong Activations

How Neuronal Noises Influence the Spiking Neural Networks’s Cognitive Learning Process: A Preliminary Study