Approximation Problems In Sigma-Pi Neural Networks

Chunming Li,Tianping Chen,李纯明,陈天平

1996-01-01

Chinese Science Bulletin

Abstract:null

What problem does this paper attempt to address?

Approximation Capability of a Novel Neural Network Model for Dynamic Systems

Jianhai Zhang,Wanzeng Kong,Senlin Zhang,Meiqin Liu

DOI: https://doi.org/10.1109/icicta.2009.23

2009-01-01

Abstract:The approximation power for dynamic systems of a novel neural network model-standard neural network model (SNNM) is examined. Applying Stone-Weierstrass theorem, it is proved that SNNM is capable of approximating dynamic systems to any degree of accuracy. Furthermore, the results are briefly extended for any bounded measurable functions. The approximation capability together with the learn ability justify the use of SNNM in practical applications.
Novel &Lt;inline-Formula> &Lt;tex-Math Notation="tex">${\rm S}\alpha{\rm S} $&Lt;/tex-Math></inline-formula> PDF Approximations and Their Applications in Wireless Signal Detection

Yunfei Chen,Jiming Chen

DOI: https://doi.org/10.1109/twc.2014.2364181

IF: 10.4

2015-01-01

IEEE Transactions on Wireless Communications

Abstract:Three new approximations to the probability density function (PDF) of the symmetric alpha stable (SαS) distribution are proposed. The first two approximations use rational functions while the third approximation uses power functions. Using these approximations, new detectors for signals in symmetric alpha stable noise are also derived. Numerical results show that all these new approximations have good accuracies. Numerical results also show that the new detectors based on these approximations outperform the existing detectors, especially when the characteristic exponent of the symmetric alpha stable distribution is small.
Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Zuowei Shen,Haizhao Yang,Shijun Zhang

DOI: https://doi.org/10.48550/arXiv.2107.02397

2022-09-26

Abstract:This paper develops simple feed-forward neural networks that achieve the universal approximation property for all continuous functions with a fixed finite number of neurons. These neural networks are simple because they are designed with a simple, computable, and continuous activation function $\sigma$ leveraging a triangular-wave function and the softsign function. We first prove that $\sigma$-activated networks with width $36d(2d+1)$ and depth $11$ can approximate any continuous function on a $d$-dimensional hypercube within an arbitrarily small error. Hence, for supervised learning and its related regression problems, the hypothesis space generated by these networks with a size not smaller than $36d(2d+1)\times 11$ is dense in the continuous function space $C([a,b]^d)$ and therefore dense in the Lebesgue spaces $L^p([a,b]^d)$ for $p\in [1,\infty)$. Furthermore, we show that classification functions arising from image and signal classification are in the hypothesis space generated by $\sigma$-activated networks with width $36d(2d+1)$ and depth $12$ when there exist pairwise disjoint bounded closed subsets of $\mathbb{R}^d$ such that the samples of the same class are located in the same subset. Finally, we use numerical experimentation to show that replacing the rectified linear unit (ReLU) activation function by ours would improve the experiment results.

Machine Learning
An approximation by neural networkswith a fixed weight

Nahmwoo Hahm,Bum Il Hong

DOI: https://doi.org/10.1016/j.camwa.2003.06.008

2004-06-01

Abstract:Neural networks are widely used in many applications including astronomical physics,image processing, recognition, robotics, and automated target tracking, etc. Their ability to approximate arbitrary functions is the main reason for this popularity. In this paper, we discuss the constructive approximation on the whole real line by a neural networks with a sigmoidal activation function and a fixed weight. Using the convolution method, we show neural network approximation with a fixed weight to a continuous function on a compact interval. Also, we demonstrate a computational work that shows good agreement with theory.

mathematics, applied
Some queries on (quote)Comments on 'Approximation capability in C(R~/sup n/) by multilayer feedforward networks and related problems'(quote)

Tianping Chen

DOI: https://doi.org/10.1109/72.363453

2001-01-01

IEEE Transactions on Neural Networks

Abstract:In this paper, we investigate the capability of approximating functions in C(R (n)) by three-layered neural networks with sigmoidal function in the hidden layer. It is found that the boundedness condition on the sigmoidal function plays an essential role in the approximation, as contrast to continuity or monotonity condition. We point out that in order to prove the neural network in the n-dimensional case, all one needs to do is to prove the case for one dimension. The approximation in L(p)-norm (1
Deep Neural Network Approximation of Composition Functions: with Application to PINNs

Chenguang Duan,Yuling Jiao,Xiliang Lu,Jerry Zhijian Yang,Cheng Yuan

DOI: https://doi.org/10.48550/arxiv.2304.07947

2023-01-01

Abstract:In this paper, we focus on approximating a natural class of functions that are compositions of smooth functions. Unlike the low-dimensional support assumption on the covariate, we demonstrate that composition functions have an intrinsic sparse structure if we assume each layer in the composition has a small degree of freedom. This fact can alleviate the curse of dimensionality in approximation errors by neural networks. Specifically, by using mathematical induction and the multivariate Faa di Bruno formula, we extend the approximation theory of deep neural networks to the composition functions case. Furthermore, combining recent results on the statistical error of deep learning, we provide a general convergence rate analysis for the PINNs method in solving elliptic equations with compositional solutions. We also present two simple illustrative numerical examples to demonstrate the effect of the intrinsic sparse structure in regression and solving PDEs.
Some Queries on "comments on 'approximation Capability in C(R~/sup N/) by Multilayer Feedforward Networks and Related Problems'"

TP Chen

DOI: https://doi.org/10.1109/72.950157

2001-01-01

IEEE Transactions on Neural Networks

Abstract:In the comments letter by Huang et al. (ibid. vol.9 (1998)) the authors claimed that the boundedness of a sigmoidal function σ(x) is neither sufficient nor necessary condition for the validity of the approximation theorem discussed in our original paper (ibid. vol.6 (1995)). In this paper, we show that this claim is incorrect
Fuzzy RBF Neural Network Based Function Approximation

LIU Kun,YAN Gangfeng

DOI: https://doi.org/10.3969/j.issn.1000-3428.2001.02.028

2001-01-01

Abstract:In this paper we present a fuzzy RBF neural network in which the knowledge expression of fuzzy logic is combined with fast learning speed and generalization of RBF neural network. Its structure and parameters can be adjusted according to the real problem. Simulation shows its fast learning speed and approximation accuracy.
Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Li Liu,Tengchao Yu,Heng Yong

2024-09-25

Abstract:The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: \textbf{Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured?} Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of \textit{$\epsilon$ outer measure} and \textit{Numerical Span Dimension (NSdim)} to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.

Machine Learning
Approximation Accuracy of Table Look-Up Scheme for Fuzzy-Neural Networks with Bell Membership Function

Weimin Ma,Guoqing Chen

DOI: https://doi.org/10.1007/11893295_112

2006-01-01

Abstract:Although many works have been done in recent years for the designing Fuzzy-Neural Networks (FNN) from input-output data, the results concerning how to analyze the performance of some methods from a rigorous mathematical point of view are somewhat few. In this paper, the approximation bound for the Table Look-up Scheme with the Bell Membership Function is established. The detailed formulas of the error bound between the nonlinear function to be approximated and the FNN system designed based on the input-output data are derived.
A neural network for constrained saddle point problems: an approximation approach

Xisheng Shen,Shiji Song,Lixin Cheng

DOI: https://doi.org/10.1007/11539087_60

2005-01-01

Abstract:This paper proposes a neural network for saddle point problems (SPP) by an approximation approach. It first proves both the existence and the convergence property of approximate solutions, and then shows that the proposed network is globally exponentially stable and the solution of (SPP) is approximated. Simulation results are given to demonstrate further the effectiveness of the proposed network.
Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

Yunfei Yang,Zhen Li,Yang Wang

DOI: https://doi.org/10.1016/j.neunet.2022.06.013

IF: 7.8

2022-01-01

Neural Networks

Abstract:We study the expressive power of deep ReLU neural networks for approximatingfunctions in dilated shift-invariant spaces, which are widely used in signalprocessing, image processing, communications and so on. Approximation errorbounds are estimated with respect to the width and depth of neural networks.The network construction is based on the bit extraction and data-fittingcapacity of deep neural networks. As applications of our main results, theapproximation rates of classical function spaces such as Sobolev spaces andBesov spaces are obtained. We also give lower bounds of the L^p (1≤ p ≤∞) approximation error for Sobolev spaces, which show that ourconstruction of neural network is asymptotically optimal up to a logarithmicfactor.
A Priori Estimation of the Approximation, Optimization and Generalization Errors of Random Neural Networks for Solving Partial Differential Equations

Xianliang Xu,Zhongyi Huang

2024-08-17

Abstract:In recent years, neural networks have achieved remarkable progress in various fields and have also drawn much attention in applying them on scientific problems. A line of methods involving neural networks for solving partial differential equations (PDEs), such as Physics-Informed Neural Networks (PINNs) and the Deep Ritz Method (DRM), has emerged. Although these methods outperform classical numerical methods in certain cases, the optimization problems involving neural networks are typically non-convex and non-smooth, which can result in unsatisfactory solutions for PDEs. In contrast to deterministic neural networks, the hidden weights of random neural networks are sampled from some prior distribution and only the output weights participate in training. This makes training much simpler, but it remains unclear how to select the prior distribution. In this paper, we focus on Barron type functions and approximate them under Sobolev norms by random neural networks with clear prior distribution. In addition to the approximation error, we also derive bounds for the optimization and generalization errors of random neural networks for solving PDEs when the solutions are Barron type functions.

Numerical Analysis
A new Sigma-Pi-Sigma neural network based on $ L_1 $ and $ L_2 $ regularization and applications

Jianwei Jiao,Keqin Su

DOI: https://doi.org/10.3934/math.2024293

2024-01-01

AIMS Mathematics

Abstract:As one type of the important higher-order neural networks developed in the last decade, the Sigma-Pi-Sigma neural network has more powerful nonlinear mapping capabilities compared with other popular neural networks. This paper is concerned with a new Sigma-Pi-Sigma neural network based on a $ L_1 $ and $ L_2 $ regularization batch gradient method, and the numerical experiments for classification and regression problems prove that the proposed algorithm is effective and has better properties comparing with other classical penalization methods. The proposed model combines the sparse solution tendency of $ L_1 $ norm and the high benefits in efficiency of the $ L_2 $ norm, which can regulate the complexity of a network and prevent overfitting. Also, the numerical oscillation, induced by the non-differentiability of $ L_1 $ plus $ L_2 $ regularization at the origin, can be eliminated by a smoothing technique to approximate the objective function.

mathematics, applied
Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuits

Zhan Yu,Qiuhao Chen,Yuling Jiao,Yinan Li,Xiliang Lu,Xin Wang,Jerry Zhijian Yang

2024-10-09

Abstract:Parameterized quantum circuits (PQCs) have emerged as a promising approach for quantum neural networks. However, understanding their expressive power in accomplishing machine learning tasks remains a crucial question. This paper investigates the expressivity of PQCs for approximating general multivariate function classes. Unlike previous Universal Approximation Theorems for PQCs, which are either nonconstructive or rely on parameterized classical data processing, we explicitly construct data re-uploading PQCs for approximating multivariate polynomials and smooth functions. We establish the first non-asymptotic approximation error bounds for these functions in terms of the number of qubits, quantum circuit depth, and number of trainable parameters. Notably, we demonstrate that for approximating functions that satisfy specific smoothness criteria, the quantum circuit size and number of trainable parameters of our proposed PQCs can be smaller than those of deep ReLU neural networks. We further validate the approximation capability of PQCs through numerical experiments. Our results provide a theoretical foundation for designing practical PQCs and quantum neural networks for machine learning tasks that can be implemented on near-term quantum devices, paving the way for the advancement of quantum machine learning.

Quantum Physics,Machine Learning
Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems

Wei Kang,Qi Gong

DOI: https://doi.org/10.48550/arXiv.2012.01698

2020-12-03

Abstract:As demonstrated in many areas of real-life applications, neural networks have the capability of dealing with high dimensional data. In the fields of optimal control and dynamical systems, the same capability was studied and verified in many published results in recent years. Towards the goal of revealing the underlying reason why neural networks are capable of solving some high dimensional problems, we develop an algebraic framework and an approximation theory for compositional functions and their neural network approximations. The theoretical foundation is developed in a way so that it supports the error analysis for not only functions as input-output relations, but also numerical algorithms. This capability is critical because it enables the analysis of approximation errors for problems for which analytic solutions are not available, such as differential equations and optimal control. We identify a set of key features of compositional functions and the relationship between the features and the complexity of neural networks. In addition to function approximations, we prove several formulae of error upper bounds for neural networks that approximate the solutions to differential equations, optimization, and optimal control.

Machine Learning,Neural and Evolutionary Computing,Numerical Analysis
Optimal Rates of Approximation by Shallow ReLU Neural Networks and Applications to Nonparametric Regression

Yunfei Yang,Ding-Xuan Zhou

DOI: https://doi.org/10.1007/s00365-024-09679-z

2024-02-28

Constructive Approximation

Abstract:We study the approximation capacity of some variation spaces corresponding to shallow ReLU neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning Hölder functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.

mathematics
A Novel Approximation Methodology and Its Efficient VLSI Implementation for the Sigmoid Function.

Zidi Qin,Yuou Qiu,Huaqing Sun,Zhonghai Lu,Zhongfeng Wang,Qinghong Shen,Hongbing Pan

DOI: https://doi.org/10.1109/tcsii.2020.2999458

2020-01-01

IEEE Transactions on Circuits & Systems II Express Briefs

Abstract:In this brief, a novel approximation method and its optimized hardware implementation are proposed for the sigmoid function used in Deep Neural Networks (DNNs). Based on piecewise approximation and truncated Taylor series expansion, the proposed method achieves very good approximation with low complexity while exploiting data representation with powers of two. In addition, by analyzing gradients of the sigmoid function, a small trick is introduced to improve the approximation precision. Furthermore, to reduce the hardware complexity and shorten the critical path, sampled values of the function are generated with simple logical-mapping. It is shown that the proposed approximation schemes can be implemented with purely combinational logic and the sigmoid function can be computed in one clock cycle. The experimental results demonstrate that the mean absolute errors are at the order of 1 × 10 -3 . Compared with prior arts, the new design can obtain significant improvement in critical path with comparable performance.
Neural Network Approximators for Marginal MAP in Probabilistic Circuits

Shivvrat Arya,Tahrima Rahman,Vibhav Gogate

2024-02-06

Abstract:Probabilistic circuits (PCs) such as sum-product networks efficiently represent large multi-variate probability distributions. They are preferred in practice over other probabilistic representations such as Bayesian and Markov networks because PCs can solve marginal inference (MAR) tasks in time that scales linearly in the size of the network. Unfortunately, the maximum-a-posteriori (MAP) and marginal MAP (MMAP) tasks remain NP-hard in these models. Inspired by the recent work on using neural networks for generating near-optimal solutions to optimization problems such as integer linear programming, we propose an approach that uses neural networks to approximate (M)MAP inference in PCs. The key idea in our approach is to approximate the cost of an assignment to the query variables using a continuous multilinear function, and then use the latter as a loss function. The two main benefits of our new method are that it is self-supervised and after the neural network is learned, it requires only linear time to output a solution. We evaluate our new approach on several benchmark datasets and show that it outperforms three competing linear time approximations, max-product inference, max-marginal inference and sequential estimation, which are used in practice to solve MMAP tasks in PCs.

Machine Learning,Artificial Intelligence
INA: Incremental Network Approximation Algorithm for Limited Precision Deep Neural Networks

Zheyu Liu,Kaige Jia,Weiqiang Liu,Qi Wei,Fei Qiao,Huazhong Yang

DOI: https://doi.org/10.1109/iccad45719.2019.8942054

2019-11-01

Abstract:Approximate computing is a promising paradigm to deal with large computing workloads in fault-tolerant applications, providing opportunities to improve hardware efficiency of Deep Neural Networks (DNNs). However, it is still difficult to apply highly approximate arithmetics (e.g., multipliers) to DNNs due to the effect of error accumulation and the convergence problem in re-training phase. To tackle this limitation, we propose a hardware-software co-design algorithm, namely Incremental Network Approximation (INA). By addressing the convergence problem, INA promotes fault tolerance of DNNs, and yields more tradeoffs between accuracy and implementation cost. Experiments show that the approximate inference models re-trained by INA could achieve up to 80% hardware reduction in various hardware design level, while the classification accuracy degradation is less than 2%. Moreover, the experiments also exhibit the generality of INA algorithm for applying to various approximate multiplier design.

Approximation Problems In Sigma-Pi Neural Networks

Approximation Capability of a Novel Neural Network Model for Dynamic Systems

Novel &Lt;inline-Formula&gt; &Lt;tex-Math Notation="tex"&gt;${\rm S}\alpha{\rm S} $&Lt;/tex-Math&gt;&lt;/inline-formula&gt; PDF Approximations and Their Applications in Wireless Signal Detection

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

An approximation by neural networkswith a fixed weight

Some queries on (quote)Comments on 'Approximation capability in C(R~/sup n/) by multilayer feedforward networks and related problems'(quote)

Deep Neural Network Approximation of Composition Functions: with Application to PINNs

Some Queries on "comments on 'approximation Capability in C(R~/sup N/) by Multilayer Feedforward Networks and Related Problems'"

Fuzzy RBF Neural Network Based Function Approximation

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

Approximation Accuracy of Table Look-Up Scheme for Fuzzy-Neural Networks with Bell Membership Function

A neural network for constrained saddle point problems: an approximation approach

Approximation in Shift-Invariant Spaces with Deep ReLU Neural Networks

A Priori Estimation of the Approximation, Optimization and Generalization Errors of Random Neural Networks for Solving Partial Differential Equations

A new Sigma-Pi-Sigma neural network based on $ L_1 $ and $ L_2 $ regularization and applications

Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuits

Neural Network Approximations of Compositional Functions With Applications to Dynamical Systems

Optimal Rates of Approximation by Shallow ReLU Neural Networks and Applications to Nonparametric Regression

A Novel Approximation Methodology and Its Efficient VLSI Implementation for the Sigmoid Function.

Neural Network Approximators for Marginal MAP in Probabilistic Circuits

INA: Incremental Network Approximation Algorithm for Limited Precision Deep Neural Networks

Novel &Lt;inline-Formula> &Lt;tex-Math Notation="tex">${\rm S}\alpha{\rm S} $&Lt;/tex-Math></inline-formula> PDF Approximations and Their Applications in Wireless Signal Detection