Abstract:The foundations of deep learning are supported by the seemingly opposing perspectives of approximation or learning theory. The former advocates for large/expressive models that need not generalize, while the latter considers classes that generalize but may be too small/constrained to be universal approximators. Motivated by real-world deep learning implementations that are both expressive and statistically reliable, we ask: "Is there a class of neural networks that is both large enough to be universal but structured enough to generalize?" This paper constructively provides a positive answer to this question by identifying a highly structured class of ReLU multilayer perceptions (MLPs), which are optimal function approximators and are statistically well-behaved. We show that any $L$-Lipschitz function from $[0,1]^d$ to $[-n,n]$ can be approximated to a uniform $Ld/(2n)$ error on $[0,1]^d$ with a sparsely connected $L$-Lipschitz ReLU MLP of width $\mathcal{O}(dn^d)$, depth $\mathcal{O}(\log(d))$, with $\mathcal{O}(dn^d)$ nonzero parameters, and whose weights and biases take values in $\{0,\pm 1/2\}$ except in the first and last layers which instead have magnitude at-most $n$. Unlike previously known "large" classes of universal ReLU MLPs, the empirical Rademacher complexity of our class remains bounded even when its depth and width become arbitrarily large. Further, our class of MLPs achieves a near-optimal sample complexity of $\mathcal{O}(\log(N)/\sqrt{N})$ when given $N$ i.i.d. normalized sub-Gaussian training samples. We achieve this by avoiding the standard approach to constructing optimal ReLU approximators, which sacrifices regularity by relying on small spikes. Instead, we introduce a new construction that perfectly fits together linear pieces using Kuhn triangulations and avoids these small spikes.

Efficient Estimation of Multidimensional Regression Model with Multilayer Perceptron

Efficient Estimation of Multidimensional Regression Model using Multilayer Perceptrons

An efficient multilayer RBF neural network and its application to regression problems

A Smooth Optimisation Perspective on Designing and Training Feedforward Multilayer Perceptrons

Implementing the ICE Estimator in Multilayer Perceptron Classifiers

Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks

Nonparametric regression using deep neural networks with ReLU activation function

Multi-Model Least Squares-Based Recomputation Framework for Large Data Analysis

Multiple Output Regression with Latent Noise

Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensions

Bridging the Gap Between Approximation and Learning via Optimal Approximation by ReLU MLPs of Maximal Regularity

A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability

Efficient Batch and Recursive Least Squares for Matrix Parameter Estimation

Penalized Sparse Covariance Regression with High Dimensional Covariates

Orthogonal Matching Pursuit for Multilayer Perceptions Neural Networks Model Reduction.

Calibrated Multivariate Regression Networks

Nearest Neighbors for Matrix Estimation Interpreted as Blind Regression for Latent Variable Model

Low-rank matrix estimation via nonconvex optimization methods in multi-response errors-in-variables regression

A novel Mathematical Modeling for Deep Multilayer Perceptron Optimization: Architecture Optimization and Activation Functions Selection

Large covariance matrix estimation via penalized log-det heuristics

Penalized Maximum Likelihood Estimation of Multi-layered Gaussian Graphical Models