Abstract:Disobeying the classical wisdom of statistical learning theory, modern deep neural networks generalize well even though they typically contain millions of parameters. Recently, it has been shown that the trajectories of iterative optimization algorithms can possess fractal structures, and their generalization error can be formally linked to the complexity of such fractals. This complexity is measured by the fractal's intrinsic dimension, a quantity usually much smaller than the number of parameters in the network. Even though this perspective provides an explanation for why overparametrized networks would not overfit, computing the intrinsic dimension (e.g., for monitoring generalization during training) is a notoriously difficult task, where existing methods typically fail even in moderate ambient dimensions. In this study, we consider this problem from the lens of topological data analysis (TDA) and develop a generic computational tool that is built on rigorous mathematical foundations. By making a novel connection between learning theory and TDA, we first illustrate that the generalization error can be equivalently bounded in terms of a notion called the 'persistent homology dimension' (PHD), where, compared with prior work, our approach does not require any additional geometrical or statistical assumptions on the training dynamics. Then, by utilizing recently established theoretical results and TDA tools, we develop an efficient algorithm to estimate PHD in the scale of modern deep neural networks and further provide visualization tools to help understand generalization in deep learning. Our experiments show that the proposed approach can efficiently compute a network's intrinsic dimension in a variety of settings, which is predictive of the generalization error.

Topological Measurement of Deep Neural Networks Using Persistent Homology

Experimental Observations of the Topology of Convolutional Neural Network Activations

Topological Data Analysis of Neural Network Layer Representations

Learning Persistent Homology of 3D Point Clouds

A Comprehensive Review of Deep Neural Network Interpretation Using Topological Data Analysis

Topological Neural Networks go Persistent, Equivariant, and Continuous

Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

Understanding Deep Neural Networks Using Topological Data Analysis

On Characterizing the Evolution of Embedding Space of Neural Networks using Algebraic Topology

Functional Loops: Monitoring Functional Organization of Deep Neural Networks Using Algebraic Topology

Characterization of topological structures in different neural network architectures

Be Persistent: Towards a Unified Solution for Mitigating Shortcuts in Deep Learning

Representing the Topology of Complex Networks Based on Graph Embedding

Topological Data Analysis for Multivariate Time Series Data

Learning Topological Representations for Deep Image Understanding

Deep Neural Networks via Complex Network Theory: a Perspective

Image Classification using Combination of Topological Features and Neural Networks

An Algorithm for Persistent Homology Computation Using Homomorphic Encryption

Activation Landscapes as a Topological Summary of Neural Network Performance

Architectures of Topological Deep Learning: A Survey of Message-Passing Topological Neural Networks

Topological exploration of artificial neuronal network dynamics