Chebyshev Feature Neural Network for Accurate Function Approximation

Zhongshu Xu,Yuan Chen,Dongbin Xiu

2024-09-28

Abstract:We present a new Deep Neural Network (DNN) architecture capable of approximating functions up to machine accuracy. Termed Chebyshev Feature Neural Network (CFNN), the new structure employs Chebyshev functions with learnable frequencies as the first hidden layer, followed by the standard fully connected hidden layers. The learnable frequencies of the Chebyshev layer are initialized with exponential distributions to cover a wide range of frequencies. Combined with a multi-stage training strategy, we demonstrate that this CFNN structure can achieve machine accuracy during training. A comprehensive set of numerical examples for dimensions up to $20$ are provided to demonstrate the effectiveness and scalability of the method.

Machine Learning,Neural and Evolutionary Computing,Numerical Analysis

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the accuracy of deep neural networks (DNN) in function approximation, especially for scientific computing problems. Specifically, the existing DNN can only reach a certain level of accuracy (such as $O(10^{-5})$ to $O(10^{-2})$) in many scientific computing problems, which is not sufficient for long - term prediction tasks that require high accuracy. To solve this problem, the author proposes the Chebyshev Feature Neural Network (CFNN) architecture, aiming to achieve machine - precision - level function approximation through the following improvements: 1. **Using Chebyshev features with learnable frequencies**: Introduce Chebyshev functions in the first hidden layer. These functions have superior function approximation performance and contain fewer hyper - parameters compared to Fourier feature networks. 2. **Multi - stage training strategy**: Gradually approximate the target function through a multi - stage training method. Each stage is trained based on the residual of the previous stage, thereby gradually improving the accuracy. 3. **Exponential distribution initialization**: Randomly initialize the Chebyshev frequency parameters using an exponential distribution to cover a wide frequency range and alleviate the difficulty of selecting frequency parameters. Through these improvements, CFNN can reach machine precision during the training process and is suitable for smooth and non - smooth function approximation in various dimensions. The effectiveness and scalability of CFNN are verified through a series of numerical experiments in the paper.

Chebyshev Feature Neural Network for Accurate Function Approximation

HCFNN: High-order Coverage Function Neural Network for Image Classification

Multi-stage Neural Networks: Function Approximator of Machine Precision

Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neurons

Why Deep Neural Networks for Function Approximation?

Fuzzy RBF Neural Network Based Function Approximation

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

Chebyshev Polynomial-Based Kolmogorov-Arnold Networks: An Efficient Architecture for Nonlinear Function Approximation

Optimal Neural Network Approximation for High-Dimensional Continuous Functions

ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units via Chebyshev Approximation

Gradient-enhanced deep neural network approximations

ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units via Chebyshev Approximations

Deep neural network approximation of composite functions without the curse of dimensionality

Convolutional Neural Networks on Graphs with Chebyshev Approximation, Revisited

Fourier Neural Networks for Function Approximation

Approximation smooth and sparse functions by deep neural networks without saturation

Efficient and Accurate Approximations of Nonlinear Convolutional Networks

Self-adaptive deep neural network: Numerical approximation to functions and PDEs

Approximation analysis of CNNs from a feature extraction view

Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions

Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations