Abstract:We prove that a classifier with a Barron-regular decision boundary can be approximated with a rate of high polynomial degree by ReLU neural networks with three hidden layers when a margin condition is assumed. In particular, for strong margin conditions, high-dimensional discontinuous classifiers can be approximated with a rate that is typically only achievable when approximating a low-dimensional smooth function. We demonstrate how these expression rate bounds imply fast-rate learning bounds that are close to $n^{-1}$ where $n$ is the number of samples. In addition, we carry out comprehensive numerical experimentation on binary classification problems with various margins. We study three different dimensions, with the highest dimensional problem corresponding to images from the MNIST data set.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to solve the problem of decision - boundary approximation in high - dimensional classification problems, especially when the margin condition is met. Specifically, the author explores how to use deep neural networks (DNNs) with ReLU activation functions to approximate classifiers with Barron - regular decision boundaries under the following conditions:
1. **Classification problems in high - dimensional space**: In high - dimensional space, classification problems are usually very complex because the increase in data dimensions leads to the "curse of dimensionality", making traditional smoothness assumptions no longer applicable.
2. **Discontinuous classifiers**: The research object is the discontinuous indicator function \(1_\Omega(x)\), which is difficult to approximate by traditional methods, especially in high - dimensional cases.
3. **Margin Condition**: By introducing the margin condition, that is, the distance from data points to the decision boundary has a certain distribution law, the distribution density of data points near the decision boundary can be controlled, thereby simplifying the learning task.
4. **Barron - regular decision boundaries**: It is assumed that the decision boundaries can be described by Barron - class functions. These functions have good approximation properties and are especially suitable for approximation by shallow ReLU neural networks.
### Main contributions
- **Approximation rate**: It is proved that under the premise of meeting the margin condition and tube compatibility, classifiers with Barron - regular decision boundaries can be approximated at a high polynomial - order speed by ReLU neural networks with three hidden layers. In particular, for strong margin conditions, high - dimensional discontinuous classifiers can be approximated at a rate similar to that of low - dimensional smooth functions.
- **Fast learning rate**: It shows how these expression rate bounds imply fast learning rate bounds, approaching a rate of \(n^{-1}\), where \(n\) is the number of samples.
- **Numerical experiments**: Comprehensive numerical experiments have been carried out to verify the effectiveness of the theoretical results in different dimensions (including the image classification problem of the MNIST dataset). Experiments show that as the margin index \(\gamma\) and the number of samples \(n\) increase, the test error does indeed decrease at a rate close to the theoretical prediction.
### Conclusion
By combining the margin condition, Barron - regularity and the characteristics of neural networks, this paper provides an effective method for dealing with high - dimensional classification problems and shows that near - optimal learning rates can be obtained under specific conditions. This provides a new theoretical basis and practical guidance for understanding and improving high - dimensional data classification.