Abstract:We explain how to use Kolmogorov Superposition Theorem (KST) to break the curse of dimensionality when approximating a dense class of multivariate continuous functions. We first show that there is a class of functions called $K$-Lipschitz continuous in $C([0,1]^d)$ which can be approximated by a special ReLU neural network of two hidden layers with a dimension independent approximation rate $O(n^{-1})$ with approximation constant increasing quadratically in $d$. The number of parameters used in such neural network approximation equals to $(6d+2)n$. Next we introduce KB-splines by using linear B-splines to replace the K-outer function and smooth the KB-splines to have the so-called LKB-splines as the basis for approximation. Our numerical evidence shows that the curse of dimensionality is broken in the following sense: When using the standard discrete least squares (DLS) method to approximate a continuous function, there exists a pivotal set of points in $[0,1]^d$ with size at most $O(nd)$ such that the rooted mean squares error (RMSE) from the DLS based on the pivotal set is similar to the RMSE of the DLS based on the original set with size $O(n^d)$. In addition, by using matrix cross approximation technique, the number of LKB-splines used for approximation is the same as the size of the pivotal data set. Therefore, we do not need too many basis functions as well as too many function values to approximate a high dimensional continuous function $f$.

Approximation capability of two hidden layer feedforward neural networks with fixed weights

On the approximation by single hidden layer feedforward neural networks with fixed weights

Approximation error of single hidden layer neural networks with fixed weights

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

An approximation by neural networkswith a fixed weight

Measure theoretic results for approximation by neural networks with limited weights

Function Approximation by Random Neural Networks with a Bounded Number of Layers

Approximation by random networks with bounded number of layers

Approximation Theory of Tree Tensor Networks: Tensorized Multivariate Functions

Efficient uniform approximation using Random Vector Functional Link networks

Random Vector Functional Link Networks for Function Approximation on Manifolds

Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

Approximation Theory of Tree Tensor Networks: Tensorized Univariate Functions -- Part II

Chebyshev Feature Neural Network for Accurate Function Approximation

Why Deep Neural Networks for Function Approximation?

Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class.

The Kolmogorov Superposition Theorem can Break the Curse of Dimensionality When Approximating High Dimensional Functions

Fredholm integral equations for function approximation and the training of neural networks

Function Approximation with Randomly Initialized Neural Networks for Approximate Model Reference Adaptive Control

Approximation of Time-Frequency Shift Equivariant Maps by Neural Networks