Abstract:The foundations of deep learning are supported by the seemingly opposing perspectives of approximation or learning theory. The former advocates for large/expressive models that need not generalize, while the latter considers classes that generalize but may be too small/constrained to be universal approximators. Motivated by real-world deep learning implementations that are both expressive and statistically reliable, we ask: "Is there a class of neural networks that is both large enough to be universal but structured enough to generalize?" This paper constructively provides a positive answer to this question by identifying a highly structured class of ReLU multilayer perceptions (MLPs), which are optimal function approximators and are statistically well-behaved. We show that any $L$-Lipschitz function from $[0,1]^d$ to $[-n,n]$ can be approximated to a uniform $Ld/(2n)$ error on $[0,1]^d$ with a sparsely connected $L$-Lipschitz ReLU MLP of width $\mathcal{O}(dn^d)$, depth $\mathcal{O}(\log(d))$, with $\mathcal{O}(dn^d)$ nonzero parameters, and whose weights and biases take values in $\{0,\pm 1/2\}$ except in the first and last layers which instead have magnitude at-most $n$. Unlike previously known "large" classes of universal ReLU MLPs, the empirical Rademacher complexity of our class remains bounded even when its depth and width become arbitrarily large. Further, our class of MLPs achieves a near-optimal sample complexity of $\mathcal{O}(\log(N)/\sqrt{N})$ when given $N$ i.i.d. normalized sub-Gaussian training samples. We achieve this by avoiding the standard approach to constructing optimal ReLU approximators, which sacrifices regularity by relying on small spikes. Instead, we introduce a new construction that perfectly fits together linear pieces using Kuhn triangulations and avoids these small spikes.

Universal Approximation with Convex Optimization: Gimmick or Reality? [discussion Forum].

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Extreme Learning Machine with Multiple Kernels

Efficient Convex Algorithms for Universal Kernel Learning

Designing Universally-Approximating Deep Neural Networks: A First-Order Optimization Approach

A Universal Approximation Result for Difference of log-sum-exp Neural Networks

Adaptivity and Optimality: A Universal Algorithm for Online Convex Optimization

Universal Neural Functionals

Bridging the Gap Between Approximation and Learning via Optimal Approximation by ReLU MLPs of Maximal Regularity

Learning Convex Optimization Models

Universal Approximation of Multiple Nonlinear Operators by Neural Networks

Dynamic Universal Approximation Theory: The Basic Theory for Deep Learning-Based Computer Vision Models

Optimization-Inspired Learning with Architecture Augmentations and Control Mechanisms for Low-Level Vision

Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization

Unnatural Algorithms in Machine Learning

Efficient Model-Based Concave Utility Reinforcement Learning through Greedy Mirror Descent

Universal Approximation Capability of Broad Learning System and Its Structural Variations

Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach.

Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization

Universal Approximation of Extreme Learning Machine With Adaptive Growth of Hidden Nodes.

Convergence of Expectation-Maximization Algorithm with Mixed-Integer Optimization