The errors of approximation for feedforward neural networks in the Lp metric

Feilong Cao,Rui Zhang

DOI: https://doi.org/10.1016/j.mcm.2008.07.031

2009-01-01

Mathematical and Computer Modelling

Abstract:Two classes of feedforward neural networks (FNNs) with one hidden layer are constructed to approximate L^p integrable functions in this paper. We not only show that the constructed FNNs can approximate any f@?L^p[a,b](1@?p

What problem does this paper attempt to address?

Convergence Rates for A Class of Neural Networks with Logarithmic Function

Feilong Cao,Yubo Yuan

DOI: https://doi.org/10.1109/grc.2009.5255166

2009-01-01

Abstract:The aim of this paper is to estimate the approximation error which results from the method of feed-forward neural networks (FNNs) with logarithmic sigmoidal function s(x) = (1 + e(-x))(-1). By means of an extending function approach, a class of FNNs with single hidden layer and the active function s(x) is constructed to approximate the continuous function defined on a compact interval. By using the modulus of continuity of function as metric, the rate of convergence of the FNNs is estimated. Also, a numerical examples for illustrating the theoretical results is given.
Lp Simultaneous Approximation by Neural Networks with One Hidden Layer

CAO Fei-Long,LI You-Mei,XU Zong-Ben

2003-01-01

Ruan Jian Xue Bao/Journal of Software

Abstract:It is shown in this paper by a constructive method that for any Lebesgue integrable functions defined on a compact set in a multidimensional Euclidian space, the function and its derivatives can be simultaneously approximated by a neural network with one hidden layer. This approach naturally yields the design of the hidden layer and the convergence rate. The obtained results describe the relationship between the rate of convergence of networks and the numbers of units of the hidden layer, and generalize some known density results in uniform measure.
The Estimate for Approximation Error of Neural Networks: A Constructive Approach

Feilong Cao,Tingfan Xie,Zongben Xu

DOI: https://doi.org/10.1016/j.neucom.2007.07.024

IF: 6

2007-01-01

Neurocomputing

Abstract:Neural networks are widely used in many applications including astronomical physics, image processing, recognition, robotics and automated target tracking, etc. Their ability to approximate arbitrary functions is the main reason for this popularity. The main result of this paper is a constructive proof of a formula for the upper bound of the approximation error by feedforward neural networks with one hidden layer of sigmoidal units and a linear output. The result can also be used to estimate complexity of the maximum error network. An example to demonstrate the theoretical result is given.
Constructive approximate interpolation by neural networks in the metric space

Feilong Cao,Shaobo Lin,Zongben Xu

DOI: https://doi.org/10.1016/j.mcm.2010.06.035

2010-01-01

Mathematical and Computer Modelling

Abstract:In this paper, we construct two types of feed-forward neural networks (FNNs) which can approximately interpolate, with arbitrary precision, any set of distinct data in the metric space. Firstly, for analytic activation function, an approximate interpolation FNN is constructed in the metric space, and the approximate error for this network is deduced by using Taylor formula. Secondly, for a bounded sigmoidal activation function, exact interpolation and approximate interpolation FNNs are constructed in the metric space. Also the error between the exact and approximate interpolation FNNs is given.
Characterization of Degree of Approximation for Neural Networks with One Hidden Layer

Feilong Cao,Zongben Xu,Manxi He

DOI: https://doi.org/10.1109/ICMLC.2006.259143

2006-01-01

Abstract:There have been various studies on approximation ability of feedforward neural networks (FNNs). Most of the existing studies are, however, only concerned with density or upper bound estimation on how a function can be approximated by an FNN, and consequently, the essential approximation ability of an FNN can not been revealed. In this paper, by establishing both upper and lower bound estimations on degree of approximation, the essential approximation ability of a class of FNNs is clarified in terms of the modulus of smoothness of functions to be approximated. The involved FNNs can not only approximate any continuous functions arbitrarily well, but also provide an explicit lower bound on number of hidden units required. By making use of approximation tools, it is shown that when the functions to be approximated are Lipschitzian, the approximation speed of the FNNs is determined by modulus of smoothness of the functions
Pointwise Approximation for Neural Networks

Feilong Cao,Zongben Xu,Youmei Li

DOI: https://doi.org/10.1007/11427391_5

2005-01-01

Abstract:It is shown in this paper by a constructive method that for any f ∈ C (m)[a,b], the function and its m order derivatives can be simultaneously approximated by a neural network with one hidden layer in the pointwise sense. This approach naturally yields the design of the hidden layer and the estimate of rate of convergence. The obtained results describe the relationship among the approximation degree of networks, the number of neurons in the hidden layer and the input sample, and reveal that the approximation speed of the constructed networks depends on the smoothness of approximated function.
Exact and Approximate Interpolation for Neural Networks with Single Hidden Layer

Feilong Cao,Yubo Yuan,Chunmei Ding

DOI: https://doi.org/10.1109/icnc.2010.5583824

2010-01-01

Abstract:Let φ be a bounded function on (−∞,+∞) and limx→+∞ φ(x) = M, limx→−∞ φ(x) = m, which is called general sigmoidal function. Using the general sigmoidal function as the activation function, we first construct a type of single hidden layer feedforward neural networks (FNNs) with n + 1 hidden neurons, which can learn n + 1 distinct samples with zero error. Then we present a class of FNNs with single hidden layer, namely, the approximate interpolation neural networks, which can approximately interpolate, with arbitrary precision, any set of distinct data in one dimension. Finally, we estimate the errors between the exact and approximate interpolation neural networks by means of the algebraic methods.
Approximate Interpolation by a Class of Neural Networks in Lebesgue Metric.

Chunmei Ding,Yubo Yuan,Feilong Cao

DOI: https://doi.org/10.1109/icmlc.2010.5580730

2010-01-01

Abstract:In this paper, a class of approximate interpolation neural networks is constructed to approximate Lebesgue integrable functions. It is showed that the networks can arbitrarily approximate any p-th Lebesgue integrable function in Lebesgue metric as long as the number of hidden nodes is sufficiently large. The relation among the approximation speed, the number of hidden nodes, the interpolation sample and the smoothness of the target function is also revealed by designing the Steklov mean function and the modulus of smoothness of f. The obtained results are helpful in studying the problem of approximation complexity of interpolation neural networks in Lebesgue metric.
A general approximation lower bound in $L^p$ norm, with applications to feed-forward neural networks

El Mehdi Achour,Armand Foucault,Sébastien Gerchinovitz,François Malgouyres

DOI: https://doi.org/10.48550/arXiv.2206.04360

2022-12-20

Abstract:We study the fundamental limits to the expressive power of neural networks. Given two sets $F$, $G$ of real-valued functions, we first prove a general lower bound on how well functions in $F$ can be approximated in $L^p(\mu)$ norm by functions in $G$, for any $p \geq 1$ and any probability measure $\mu$. The lower bound depends on the packing number of $F$, the range of $F$, and the fat-shattering dimension of $G$. We then instantiate this bound to the case where $G$ corresponds to a piecewise-polynomial feed-forward neural network, and describe in details the application to two sets $F$: H{ö}lder balls and multivariate monotonic functions. Beside matching (known or new) upper bounds up to log factors, our lower bounds shed some light on the similarities or differences between approximation in $L^p$ norm or in sup norm, solving an open question by DeVore et al. (2021). Our proof strategy differs from the sup norm case and uses a key probability result of Mendelson (2002).

Machine Learning,Statistics Theory
Approximation Capability of Interpolation Neural Networks

Feilong Cao,Shaobo Lin,Zongben Xu

DOI: https://doi.org/10.1016/j.neucom.2010.08.018

IF: 6

2010-01-01

Neurocomputing

Abstract:It is well-known that single hidden layer feed-forward neural networks (SLFNs) with at most n hidden neurons can learn n distinct samples with zero error, and the weights connecting the input neurons and the hidden neurons and the hidden node thresholds can be chosen randomly. Namely, for n distinct samples, there exist SLFNs with n hidden neurons that interpolate them. These networks are called exact interpolation networks for the samples. However, for some approximated target functions (as continuous or integrable functions) not all exact interpolation networks have good approximation effect. This paper, by using a functional approach, rigorously proves that for given distinct samples there exists an SLFN which not only exactly interpolates samples but also near best approximates the target function.
Some queries on (quote)Comments on 'Approximation capability in C(R~/sup n/) by multilayer feedforward networks and related problems'(quote)

Tianping Chen

DOI: https://doi.org/10.1109/72.363453

2001-01-01

IEEE Transactions on Neural Networks

Abstract:In this paper, we investigate the capability of approximating functions in C(R (n)) by three-layered neural networks with sigmoidal function in the hidden layer. It is found that the boundedness condition on the sigmoidal function plays an essential role in the approximation, as contrast to continuity or monotonity condition. We point out that in order to prove the neural network in the n-dimensional case, all one needs to do is to prove the case for one dimension. The approximation in L(p)-norm (1
Geometric Interpretation of Nonlinear Approximation Capability for Feedforward Neural Networks.

Bao-Gang Hu,Hong-Jie Xing,Yujiu Yang

DOI: https://doi.org/10.1007/978-3-540-28647-9_2

2004-01-01

Abstract:This paper presents a preliminary study on the nonlinear approximation capability of feedforward neural networks (FNNs) via a geometric approach. Three simplest FNNs with at most four free parameters are defined and investigated. By approximations on one-dimensional functions, we observe that the Chebyshev-polynomials, Gaussian, and sigmoidal FNNs are ranked in order of providing more varieties of non-linearities. If neglecting the compactness feature inherited by Gaussian neural networks, we consider that the Chebyshev-polynomial-based neural networks will be the best among three types of FNNs in an efficient use of free parameters.
Approximation Bounds by Neural Networks in L Ω P [-4Pt]

JJ Wang,ZB Xu,WJ Xu

DOI: https://doi.org/10.1007/978-3-540-28647-9_1

2004-01-01

Abstract:We consider approximation of multidimensional functions by feedforward neural networks with one hidden layer of Sigmoidal units and a linear output. Under the Orthogonal polynomials basis and certain assumptions of activation functions in the neural network, the upper bounds on the degree of approximation are obtained in the class of functions considered in this paper. The order of approximation O(n(-r/d)), d being dimension, n the number of hidden neurons, and r the natural number.
Lower Estimation of Approximation Rate for Neural Networks

FeiLong Cao,YongQuan Zhang,ZongBen Xu

DOI: https://doi.org/10.1007/s11432-009-0027-7

2009-01-01

Abstract:Let SF d and $\Pi _{\phi ,n,d} = \left\{ {\sum\nolimits_{j = 1}^n {b_j \phi (\omega _j \cdot x + \theta _j ):b_j } ,\theta _j \in \mathbb{R},\omega _j \in \mathbb{R}^d } \right\}$ be the set of periodic and Lebesgue’s square-integrable functions and the set of feedforward neural network (FNN) functions, respectively. Denote by dist (SF d Πϕ,n,d ) the deviation of the set SF d from the set Πϕ,n,d . A main purpose of this paper is to estimate the deviation. In particular, based on the Fourier transforms and the theory of approximation, a lower estimation for dist (SF d Πϕ,n,d ) is proved. That is, dist(SF d Πϕ,n,d ) ⩾ $\frac{C}{{(n\log _2 n)^{{1 \mathord{\left/ {\vphantom {1 2}} \right. \kern-\nulldelimiterspace} 2}} }}$. The obtained estimation depends only on the number of neuron in the hidden layer, and is independent of the approximated target functions and dimensional number of input. This estimation also reveals the relationship between the approximation rate of FNNs and the topology structure of hidden layer.
Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

TP CHEN,H CHEN

DOI: https://doi.org/10.1109/72.286886

1993-01-01

IEEE Transactions on Neural Networks

Abstract:The paper gives several strong results on neural network representation in an explicit form. Under very mild conditions a functional defined on a compact set in C[a, b] or Lp[a, b], spaces of infinite dimensions, can be approximated arbitrarily well by a neural network with one hidden layer. The results are a significant development beyond earlier work, where theorems of approximating continuous functions defined on a finite-dimensional real space by neural networks with one hidden layer were given. All the results are shown to be applicable to the approximation of the output of dynamic systems at any particular time
Approximation Accuracy of Table Look-Up Scheme for Fuzzy-Neural Networks with Bell Membership Function

Weimin Ma,Guoqing Chen

DOI: https://doi.org/10.1007/11893295_112

2006-01-01

Abstract:Although many works have been done in recent years for the designing Fuzzy-Neural Networks (FNN) from input-output data, the results concerning how to analyze the performance of some methods from a rigorous mathematical point of view are somewhat few. In this paper, the approximation bound for the Table Look-up Scheme with the Bell Membership Function is established. The detailed formulas of the error bound between the nonlinear function to be approximated and the FNN system designed based on the input-output data are derived.
Constructive Trigonometric Function Approximation of Neural Networks

Jianjun Wang,Zongben Xu,Jia Jing

DOI: https://doi.org/10.1109/skg.2007.17

2007-01-01

Abstract:In this paper, we consider approximation to trigonometric polynomial function by using a one-hidden-layer feedforward neural networks, and obtain the upper bounds of trigonometric function approximation by feedforward neural networks. Then we give the algorithmic example, where the networks constructed can very efficiently approximate multivariate trigonometric polynomials. The obtained results are of theoretical and practical importance in constructing a feedforward neural network with three-layer to approximate the class of multivariate trigonometric polynomials. They also provide a route in both theory and method of constructing neural network to approximate any multi- functions.
Integral transform and its application to neural network approximation

Feng-jun Li,Zongben Xu

DOI: https://doi.org/10.1007/11759966_10

2006-01-01

Abstract:Neural networks are widely used to approximate nonlinear functions. In order to study its approximation capability, a theorem of integral representation of functions is developed by using integral transform. Using the developed representation, an approximation order estimation for the bell-shaped neural networks is obtained. The obtained result reveals that the approximation accurately of the bell-shaped neural networks depends not only on the number of hidden neurons, but also on the smoothness of target functions.
Fuzzy RBF Neural Network Based Function Approximation

LIU Kun,YAN Gangfeng

DOI: https://doi.org/10.3969/j.issn.1000-3428.2001.02.028

2001-01-01

Abstract:In this paper we present a fuzzy RBF neural network in which the knowledge expression of fuzzy logic is combined with fast learning speed and generalization of RBF neural network. Its structure and parameters can be adjusted according to the real problem. Simulation shows its fast learning speed and approximation accuracy.
Approximation by approximate interpolation neural networks with single hidden layer

Chunmei Ding,Yubo Yuan,Feilong Cao

DOI: https://doi.org/10.1109/ICMLC.2010.5580832

2010-01-01

Abstract:A bounded function φ defined on (-∞, + ∞) is called general sigmoidal function if it satisfies limx→+∞φ(x) = M, limx→-∞φ(x) = m. Using the general sigmoidal function as the activation function, a type of neural networks with single hidden layer and n + 1 hidden neurons is constructed. These networks are called approximate interpolation networks, which can approximately interpolate, with arbitrary precision, any set of distinct data in one dimension. By using the modulus of continuity of function as metric, the errors of approximation by the constructed networks is estimated.

The errors of approximation for feedforward neural networks in the Lp metric

Convergence Rates for A Class of Neural Networks with Logarithmic Function

Lp Simultaneous Approximation by Neural Networks with One Hidden Layer

The Estimate for Approximation Error of Neural Networks: A Constructive Approach

Constructive approximate interpolation by neural networks in the metric space

Characterization of Degree of Approximation for Neural Networks with One Hidden Layer

Pointwise Approximation for Neural Networks

Exact and Approximate Interpolation for Neural Networks with Single Hidden Layer

Approximate Interpolation by a Class of Neural Networks in Lebesgue Metric.

A general approximation lower bound in $L^p$ norm, with applications to feed-forward neural networks

Approximation Capability of Interpolation Neural Networks

Some queries on (quote)Comments on 'Approximation capability in C(R~/sup n/) by multilayer feedforward networks and related problems'(quote)

Geometric Interpretation of Nonlinear Approximation Capability for Feedforward Neural Networks.

Approximation Bounds by Neural Networks in L Ω P [-4Pt]

Lower Estimation of Approximation Rate for Neural Networks

Approximations of Continuous Functionals by Neural Networks with Application to Dynamic Systems

Approximation Accuracy of Table Look-Up Scheme for Fuzzy-Neural Networks with Bell Membership Function

Constructive Trigonometric Function Approximation of Neural Networks

Integral transform and its application to neural network approximation

Fuzzy RBF Neural Network Based Function Approximation

Approximation by approximate interpolation neural networks with single hidden layer