Abstract:Recent years have witnessed a hot wave of deep neural networks in various domains; however, it is not yet well understood theoretically. A theoretical characterization of deep neural networks should point out their approximation ability and complexity, i.e., showing which architecture and size are sufficient to handle the concerned tasks. This work takes one step on this direction by theoretically studying the approximation and complexity of deep neural networks to invariant functions. We first prove that the invariant functions can be universally approximated by deep neural networks. Then we show that a broad range of invariant functions can be asymptotically approximated by various types of neural network models that includes the complex-valued neural networks, convolutional neural networks, and Bayesian neural networks using a polynomial number of parameters or optimization iterations. We also provide a feasible application that connects the parameter estimation and forecasting of high-resolution signals with our theoretical conclusions. The empirical results obtained on simulation experiments demonstrate the effectiveness of our method.

APPROXIMATION MULTIDIMENSION FUCTION WITH FUNCTIONAL NETWORK

Theory and Learning Algorithm of Multi-Dimensional Function Approximation Based on Functional Networks

Fuzzy RBF Neural Network Based Function Approximation

Approximation of RKHS Functionals by Neural Networks

Approximation capabilities of multilayer fuzzy neural networks on the set of fuzzy-valued functions

Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

A New Multilayer Feedforward Small-World Neural Network with Its Performances on Function Approximation

A Combined Function Approximation Model and Its Application

Integral transform and its application to neural network approximation

Approximation Capability to Functions of Several Variables, Nonlinear Functionals and Operators by Radial Basis Function Neural Networks

Function approximation ability of fuzzy neural networks

Functional Network Method for the Identification of Nonlinear Systems

Interpolation Mechanism Of Functional Networks

Investigation of Neural Networks for Function Approximation

Approximation of Functions with Spatial Inhomogeneity Based on "true" Ortho-Ridgelet Neural Network

A New Adaptive Ridgelet Neural Network

FUNCTION APPROXIMATION BASED ON COMBINED NEURAL NETWORK MODEL

Approximation to a Compact Set of Functions by Feedforward Neural Networks.

Application of Functional Networks to Solving Functional Equations

On the Approximation and Complexity of Deep Neural Networks to Invariant Functions

Approximate factorization learning algorithm of multivariate polynomials based on functional networks