Abstract:We study the expressivity of ReLU neural networks in the setting of a binary classification problem from a topological perspective. Recently, empirical studies showed that neural networks operate by changing topology, transforming a topologically complicated data set into a topologically simpler one as it passes through the layers. This topological simplification has been measured by Betti numbers, which are algebraic invariants of a topological space. We use the same measure to establish lower and upper bounds on the topological simplification a ReLU neural network can achieve with a given architecture. We therefore contribute to a better understanding of the expressivity of ReLU neural networks in the context of binary classification problems by shedding light on their ability to capture the underlying topological structure of the data. In particular the results show that deep ReLU neural networks are exponentially more powerful than shallow ones in terms of topological simplification. This provides a mathematically rigorous explanation why deeper networks are better equipped to handle complex and topologically rich data sets.

What problem does this paper attempt to address?

This paper mainly discusses the topological expressiveness of ReLU neural networks in binary classification problems. The research starts from a topological perspective, and based on recent empirical studies, it is found that neural networks transform complex datasets into simpler forms by changing the topological structure. The authors use Betti numbers as a measure and establish upper and lower bounds on the topological simplification level that ReLU neural networks can achieve. They prove that deep ReLU neural networks have exponentially stronger capability in topological simplification compared to shallow networks, providing a mathematically rigorous explanation for why deeper networks are more suitable for handling complex and topologically rich datasets. The main contributions of this paper include determining the lower and upper bounds of the topological expressiveness of ReLU neural networks, which indicate that Betti numbers can exhibit exponential growth as network depth increases. However, if the depth remains constant and the width increases, the growth of Betti numbers will be polynomial. This suggests that deep neural networks are required to accurately model datasets with high Betti numbers (parameterized by certain parameters p) when the size of the neural network is limited. In addition, the paper discusses existing work related to the expressiveness of neural networks, including empirical and theoretical studies using topological methods, as well as other metrics for neural network expressiveness, such as the number of linear regions. By comparing the difference in topological expressiveness between deep and shallow networks, the paper further supports the advantages of deep networks in handling complex tasks. In short, this paper aims to understand how ReLU neural networks capture the topological structure of data through their architecture and reveals the crucial role of depth in improving network expressiveness.

Topological Expressivity of ReLU Neural Networks

A Theoretical Study of Neural Network Expressive Power via Manifold Topology

Expressivity of ReLU-Networks under Convex Relaxations

Towards Lower Bounds on the Depth of ReLU Neural Networks

Neural networks with ReLU powers need less depth

Local and global topological complexity measures OF ReLU neural network functions

The Expressive Power of Neural Networks: A View from the Width

On the Expressive Power of Neural Networks

Deep ReLU Networks Have Surprisingly Simple Polytopes

The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks

On the Local Complexity of Linear Regions in Deep ReLU Networks

Locally linear attributes of ReLU neural networks

The Geometric Structure of Fully-Connected ReLU Layers

Expressive Power of ReLU and Step Networks under Floating-Point Operations

On Minimal Depth in Neural Networks

Beyond Counting Linear Regions of Neural Networks, Simple Linear Regions Dominate!

Topological obstruction to the training of shallow ReLU neural networks

ReLU Neural Networks, Polyhedral Decompositions, and Persistent Homolog

ReLU soothes the NTK condition number and accelerates optimization for wide neural networks

When Are Bias-Free ReLU Networks Like Linear Networks?

Exponential Expressivity of ReLU$^k$ Neural Networks on Gevrey Classes with Point Singularities