Abstract:Topological data analysis (TDA) is a branch of computational mathematics, bridging algebraic topology and data science, that provides compact, noise-robust representations of complex structures. Deep neural networks (DNNs) learn millions of parameters associated with a series of transformations defined by the model architecture, resulting in high-dimensional, difficult-to-interpret internal representations of input data. As DNNs become more ubiquitous across multiple sectors of our society, there is increasing recognition that mathematical methods are needed to aid analysts, researchers, and practitioners in understanding and interpreting how these models' internal representations relate to the final classification. In this paper, we apply cutting edge techniques from TDA with the goal of gaining insight into the interpretability of convolutional neural networks used for image classification. We use two common TDA approaches to explore several methods for modeling hidden-layer activations as high-dimensional point clouds, and provide experimental evidence that these point clouds capture valuable structural information about the model's process. First, we demonstrate that a distance metric based on persistent homology can be used to quantify meaningful differences between layers, and we discuss these distances in the broader context of existing representational similarity metrics for neural network interpretability. Second, we show that a mapper graph can provide semantic insight into how these models organize hierarchical class knowledge at each layer. These observations demonstrate that TDA is a useful tool to help deep learning practitioners unlock the hidden structures of their models.

Visualizing Deep Neural Networks with Topographic Activation Maps

Experimental Observations of the Topology of Convolutional Neural Network Activations

Understanding Neural Networks Through Deep Visualization

Understanding Deep Neural Networks Using Topological Data Analysis

Activation Landscapes as a Topological Summary of Neural Network Performance

A Comprehensive Review of Deep Neural Network Interpretation Using Topological Data Analysis

Evaluating the visualization of what a Deep Neural Network has learned

Brain Decodes Deep Nets

NeuralVis: Visualizing and Interpreting Deep Learning Models

Neural Activation Patterns (NAPs): Visual Explainability of Learned Concepts

Effective Dimensionality Reduction for Visualizing Neural Dynamics by Laplacian Eigenmaps

Deeper Interpretability of Deep Networks

Understanding CNN Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoning

Towards Utilising a Range of Neural Activations for Comprehending Representational Associations

Deep Connectomics Networks: Neural Network Architectures Inspired by Neuronal Networks

Neural Networks Decoded: Targeted and Robust Analysis of Neural Network Decisions via Causal Explanations and Reasoning

Exposition and Interpretation of the Topology of Neural Networks

Extracting knowledge from Deep Neural Networks through graph analysis

Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations

Understanding Neural Network Through Neuron Level Visualization.

Visualizing and Understanding Deep Neural Networks : A Case of Seismic Fault Detection Network