Abstract:We consider the problem of learning functions in the $\mathcal{F}_{p,\pi}$ and Barron spaces, which are natural function spaces that arise in the high-dimensional analysis of random feature models (RFMs) and two-layer neural networks. Through a duality analysis, we reveal that the approximation and estimation of these spaces can be considered equivalent in a certain sense. This enables us to focus on the easier problem of approximation and estimation when studying the generalization of both models. The dual equivalence is established by defining an information-based complexity that can effectively control estimation errors. Additionally, we demonstrate the flexibility of our duality framework through comprehensive analyses of two concrete applications. The first application is to study learning functions in $\mathcal{F}_{p,\pi}$ with RFMs. We prove that the learning does not suffer from the curse of dimensionality as long as $p>1$, implying RFMs can work beyond the kernel regime. Our analysis extends existing results [CMM21] to the noisy case and removes the requirement of overparameterization. The second application is to investigate the learnability of reproducing kernel Hilbert space (RKHS) under the $L^\infty$ metric. We derive both lower and upper bounds of the minimax estimation error by using the spectrum of the associated kernel. We then apply these bounds to dot-product kernels and analyze how they scale with the input dimension. Our results suggest that learning with ReLU (random) features is generally intractable in terms of reaching high uniform accuracy.

A Priori Generalization Error Analysis of Two-Layer Neural Networks for Solving High Dimensional Schrödinger Eigenvalue Problems

Generalization Error Estimates of Machine Learning Methods for Solving High Dimensional Schrödinger Eigenvalue Problems

Two-layers neural networks for Schr{ö}dinger eigenvalue problems

Quantitative Analysis of Generalization Ability of Neural Network with Prior-Knowledge

Deep learning neural network for approaching Schrödinger problems with arbitrary two-dimensional confinement

A Duality Framework for Generalization Analysis of Random Feature Models and Two-Layer Neural Networks.

Low regularity error estimates for high dimensional nonlinear Schrödinger equations

Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations

An L2 Analysis of Reinforcement Learning in High Dimensions with Kernel and Neural Network Approximation

A Priori Estimation of the Approximation, Optimization and Generalization Errors of Random Neural Networks for Solving Partial Differential Equations

Generalization for Least Squares Regression With Simple Spiked Covariances

Mean-field Analysis of Generalization Errors

Some observations on high-dimensional partial differential equations with Barron data

Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamics

The Efficacy of Regularization in Two Layer Neural Networks

A posteriori error estimates for Schr{ö}dinger operators discretized with linear combinations of atomic orbitals

Generalization Error of Generalized Linear Models in High Dimensions

Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory

Solving two-dimensional quantum eigenvalue problems using physics-informed machine learning