Abstract:Disobeying the classical wisdom of statistical learning theory, modern deep neural networks generalize well even though they typically contain millions of parameters. Recently, it has been shown that the trajectories of iterative optimization algorithms can possess fractal structures, and their generalization error can be formally linked to the complexity of such fractals. This complexity is measured by the fractal's intrinsic dimension, a quantity usually much smaller than the number of parameters in the network. Even though this perspective provides an explanation for why overparametrized networks would not overfit, computing the intrinsic dimension (e.g., for monitoring generalization during training) is a notoriously difficult task, where existing methods typically fail even in moderate ambient dimensions. In this study, we consider this problem from the lens of topological data analysis (TDA) and develop a generic computational tool that is built on rigorous mathematical foundations. By making a novel connection between learning theory and TDA, we first illustrate that the generalization error can be equivalently bounded in terms of a notion called the 'persistent homology dimension' (PHD), where, compared with prior work, our approach does not require any additional geometrical or statistical assumptions on the training dynamics. Then, by utilizing recently established theoretical results and TDA tools, we develop an efficient algorithm to estimate PHD in the scale of modern deep neural networks and further provide visualization tools to help understand generalization in deep learning. Our experiments show that the proposed approach can efficiently compute a network's intrinsic dimension in a variety of settings, which is predictive of the generalization error.

Does the Geometry of the Data Control the Geometry of Neural Predictions? (Student Abstract)

Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture

Information Geometry of Evolution of Neural Network Parameters While Training

A Geometric Modeling of Occam's Razor in Deep Learning

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks

Information theoretic study of the neural geometry induced by category learning

Geometry-Informed Neural Networks

Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

Neural networks learn to magnify areas near decision boundaries

Understanding over-parameterized deep networks by geometrization

Neural population geometry: An approach for understanding biological and artificial neural networks

On the Geometry of Deep Learning

The Adversarial Attack and Detection under the Fisher Information Metric

Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm

Predicting and Enhancing the Fairness of DNNs with the Curvature of Perceptual Manifolds

A Spectral Theory of Neural Prediction and Alignment

Inductive Bias of Deep Convolutional Networks through Pooling Geometry

The Geometric Occam's Razor Implicit in Deep Learning

Hardness of Learning Neural Networks under the Manifold Hypothesis

Human-Like Geometric Abstraction in Large Pre-trained Neural Networks

Fisher-Bures Adversary Graph Convolutional Networks