Abstract:We explain how to use Kolmogorov Superposition Theorem (KST) to break the curse of dimensionality when approximating a dense class of multivariate continuous functions. We first show that there is a class of functions called $K$-Lipschitz continuous in $C([0,1]^d)$ which can be approximated by a special ReLU neural network of two hidden layers with a dimension independent approximation rate $O(n^{-1})$ with approximation constant increasing quadratically in $d$. The number of parameters used in such neural network approximation equals to $(6d+2)n$. Next we introduce KB-splines by using linear B-splines to replace the K-outer function and smooth the KB-splines to have the so-called LKB-splines as the basis for approximation. Our numerical evidence shows that the curse of dimensionality is broken in the following sense: When using the standard discrete least squares (DLS) method to approximate a continuous function, there exists a pivotal set of points in $[0,1]^d$ with size at most $O(nd)$ such that the rooted mean squares error (RMSE) from the DLS based on the pivotal set is similar to the RMSE of the DLS based on the original set with size $O(n^d)$. In addition, by using matrix cross approximation technique, the number of LKB-splines used for approximation is the same as the size of the pivotal data set. Therefore, we do not need too many basis functions as well as too many function values to approximate a high dimensional continuous function $f$.

Approximation of functions from Korobov spaces by shallow neural networks

Rates of Approximation by ReLU Shallow Neural Networks

Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Norms

Approximation Rates for Shallow ReLU$^k$ Neural Networks on Sobolev Spaces via the Radon Transform

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression

Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

Why Deep Neural Networks for Function Approximation?

Optimal Rates of Approximation by Shallow ReLU Neural Networks and Applications to Nonparametric Regression

On the optimal approximation of Sobolev and Besov functions using deep ReLU neural networks

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces

Nonparametric regression using over-parameterized shallow ReLU neural networks

Uniform Approximation with Quadratic Neural Networks

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

Nearly Optimal Approximation Rates for Deep Super ReLU Networks on Sobolev Spaces

The Kolmogorov Superposition Theorem can Break the Curse of Dimensionality When Approximating High Dimensional Functions

Approximation in $L^p(μ)$ with deep ReLU neural networks

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Optimal function approximation with ReLU neural networks