Abstract:Universal approximation capability, also referred to as universality, is an important property of deep neural networks, endowing them with the potency to accurately represent the underlying target function in learning tasks. In practice, the architecture of deep neural networks largely influences the performance of the models. However, most existing methodologies for designing neural architectures, such as the heuristic manual design or neural architecture search, ignore the universal approximation property, thus losing a potential safeguard about the performance. In this paper, we propose a unified framework to design the architectures of deep neural networks with a universality guarantee based on first-order optimization algorithms, where the forward pass is interpreted as the updates of an optimization algorithm. The (explicit or implicit) network is designed by replacing each gradient term in the algorithm with a learnable module similar to a two-layer network or its derivatives Specifically, we explore the realm of width-bounded neural networks, a common practical scenario, showcasing their universality. Moreover, adding operations of normalization, downsampling, and upsampling does not hurt the universality. To the best of our knowledge, this is the first work that width-bounded networks with universal approximation guarantee can be designed in a principled way. Our framework can inspire a variety of neural architectures including some renowned structures such as ResNet and DenseNet, as well as novel innovations. The experimental results on image classification problems demonstrate that the newly inspired networks are competitive and surpass the baselines of ResNet, DenseNet, as well as the advanced ConvNeXt and ViT, testifying to the effectiveness of our framework.

Conceptual Design Optimization by Functional Neural Networks

Complex Functional Networks

Permutation Equivariant Neural Functionals

A discrete-time neural network for optimization problems with hybrid constraints.

Optimization Algorithm Inspired Deep Neural Network Structure Design

Functional Network Method for the Identification of Nonlinear Systems

Functional Network: A Novel Framework for Interpretability of Deep Neural Networks

Deep Neural Network-Based Surrogate-Assisted Inverse Optimization for High-Speed Interconnects

Combinatorial Optimization Neural Nets Based On A Hybrid Of Lagrange And Transformation Approaches

A Novel Neurodynamic Approach to Constrained Complex-Variable Pseudoconvex Optimization.

Unified Control Liapunov Function Based Design of Neural Networks That Aim at Global Minimization of Nonconvex Functions

Exploring the Complexity of Deep Neural Networks through Functional Equivalence

Max and Coincidence Neurons in Neural Networks

Neural Functional Transformers

Optimizing Neural Networks in the Equivalent Class Space.

Neural Combinatorial Optimization: a New Player in the Field

Optimizing neuronal activation function types based on GP in constructive FNN design

Design Assistant with Neural Networks

Designing Universally-Approximating Deep Neural Networks: A First-Order Optimization Approach

Optimization Proxies using Limited Labeled Data and Training Time -- A Semi-Supervised Bayesian Neural Network Approach

A Neurodynamic Approach to Nonsmooth Constrained Pseudoconvex Optimization Problem