Abstract:We perform a comprehensive numerical study of the effect of approximation-theoretical results for neural networks on practical learning problems in the context of numerical analysis. As the underlying model, we study the machine-learning-based solution of parametric partial differential equations. Here, approximation theory predicts that the performance of the model should depend only very mildly on the dimension of the parameter space and is determined by the intrinsic dimension of the solution manifold of the parametric partial differential equation. We use various methods to establish comparability between test-cases by minimizing the effect of the choice of test-cases on the optimization and sampling aspects of the learning problem. We find strong support for the hypothesis that approximation-theoretical effects heavily influence the practical behavior of learning problems in numerical analysis.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the effectiveness and feasibility of using deep neural networks (DNNs) to solve parametric partial differential equations (PPDEs) in numerical analysis. Specifically, the authors focus on how to train neural networks to approximate the solutions of parametric diffusion equations and study the performance of this method in practical applications, especially in high - dimensional parameter spaces. ### Main research objectives 1. **Identify key factors**: Determine which key characteristics of parametric problems make it easy or difficult to use neural - network - based methods for solving. 2. **Evaluate the consistency between theory and practice**: Verify whether approximate theoretical predictions can explain the behavior in actual learning problems, especially for problems in high - dimensional parameter spaces. ### Background and motivation - **Standard problems in numerical analysis**: Parametric partial differential equations are a standard problem in applied science and engineering, where certain parts (such as boundary conditions, source terms, or the shape of the domain) are controlled by a set of parameters. - **Computational challenges**: In some application scenarios, it is necessary to solve partial differential equations frequently or in real - time, and it is computationally infeasible to solve the equations separately for each parameter selection. Therefore, it is necessary to utilize the joint structure of all individual problems to reduce computational complexity. - **Machine learning methods**: In recent years, machine learning methods have been introduced to learn the mapping from parameters to solutions, replacing or enhancing traditional reduced - basis methods. ### Research methods - **Experimental design**: The authors designed a series of numerical experiments to learn the solutions of parametric diffusion equations under different parameter selections by training fully - connected neural networks. - **Controlled variables**: To ensure the comparability of experimental results, the authors took various measures, such as fixing the network architecture, analyzing the convergence behavior, and generating samples independently. ### Experimental results - **Sensitivity analysis**: The experimental results show that this method is very sensitive to different test cases, and the error between different test cases shows different qualitative behaviors with the change of the parameter space dimension. - **Absence of the curse of dimensionality**: In all test cases, the phenomenon of exponential growth of error with the parameter space dimension (i.e., the curse of dimensionality) was not observed. - **Influence of intrinsic complexity**: The error increases as the intrinsic complexity of the problem increases, indicating that the complexity of the solution manifold is an important factor affecting the performance of the method. ### Conclusions - **Generality of the method**: This method has high generality and can be applied to different types of problems. - **Consistency between theory and practice**: The experimental results are consistent with the predictions of approximate theory, supporting the practical relevance of approximate theory in deep - learning problems. ### Related work - **Early applications**: Since the 1990s, the application of neural networks in solving partial differential equations has begun to attract attention. - **Main directions**: Current research mainly focuses on learning the solutions of single PDEs, system identification, and goal - oriented methods. - **Theoretical analysis**: Many theoretical works have analyzed the approximation ability of neural networks in solving (parametric) PDEs, showing that neural networks can overcome the curse of dimensionality of certain specific PDEs. In conclusion, through detailed numerical experiments, this paper verifies the effectiveness and feasibility of using deep neural networks to solve parametric partial differential equations and explores the key factors affecting their performance.

Numerical Solution of the Parametric Diffusion Equation by Deep Neural Networks

Numerically Solving Parametric Families of High-Dimensional Kolmogorov Partial Differential Equations via Deep Learning

Solving Partial Differential Equations Using Artificial Neural Networks

Deep learning based numerical approximation algorithms for stochastic partial differential equations and high-dimensional nonlinear filtering problems

A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients

Quasi-Monte Carlo sampling for machine-learning partial differential equations

Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations

Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations

Deep-learning of parametric partial differential equations from sparse and noisy data

Space-time deep neural network approximations for high-dimensional partial differential equations

Solving parametric partial differential equations with deep rectified quadratic unit neural networks

Solving Partial Differential Equations Using Deep Learning and Physical Constraints

Neural Control of Parametric Solutions for High-dimensional Evolution PDEs

Solving parametric PDE problems with artificial neural networks

Can Physics-Informed Neural Networks beat the Finite Element Method?

A Deep Learning Neural Network Framework for Solving Singular Nonlinear Ordinary Differential Equations

Deep neural network approximations for Monte Carlo algorithms

Learning Partial Differential Equations from Data Using Neural Networks

A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations

Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks