Some Investigations on Applying Deep Learning Techniques to Solve Partial Differential Equations

Maokun Li,Tao Shan,Fan Yang,Shenheng Xuo

DOI: https://doi.org/10.1109/iceaa.2019.8879406

2019-01-01

Abstract:In this study, we investigated the feasibility of applying the deep learning techniques, more specifically, the convolutional neural network, to solve partial differential equations.

What problem does this paper attempt to address?

Solving Partial Differential Equations Using Deep Learning and Physical Constraints

Yanan Guo,Xiaoqun Cao,Bainian Liu,Mei Gao

DOI: https://doi.org/10.3390/app10175917

2020-08-26

Applied Sciences

Abstract:The various studies of partial differential equations (PDEs) are hot topics of mathematical research. Among them, solving PDEs is a very important and difficult task. Since many partial differential equations do not have analytical solutions, numerical methods are widely used to solve PDEs. Although numerical methods have been widely used with good performance, researchers are still searching for new methods for solving partial differential equations. In recent years, deep learning has achieved great success in many fields, such as image classification and natural language processing. Studies have shown that deep neural networks have powerful function-fitting capabilities and have great potential in the study of partial differential equations. In this paper, we introduce an improved Physics Informed Neural Network (PINN) for solving partial differential equations. PINN takes the physical information that is contained in partial differential equations as a regularization term, which improves the performance of neural networks. In this study, we use the method to study the wave equation, the KdV–Burgers equation, and the KdV equation. The experimental results show that PINN is effective in solving partial differential equations and deserves further research.
Study on a Poisson's Equation Solver Based on Deep Learning Technique.

Wei Tang,Tao Shan,Xunwang Dang,Maokun Li,Fan Yang,Shenheng Xu,Ji Wu

DOI: https://doi.org/10.1109/edaps.2017.8277017

2017-01-01

Abstract:In this work, we investigated the feasibility of applying deep learning techniques to solve 2D Poisson's equation. A deep convolutional neural network is set up to predict the distribution of electric potential in 2D. With training data generated from a finite difference solver, the strong approximation capability of the deep convolutional neural network allows it to make correct prediction given information of the source and distribution of permittivity. Numerical experiments show that the predication error can reach below one percent, with a significant reduction in CPU time compared with the traditional solver based on finite difference methods.
Partial Differential Equations Meet Deep Neural Networks: A Survey

Shudong Huang,Wentao Feng,Chenwei Tang,Jiancheng Lv

DOI: https://doi.org/10.48550/arXiv.2211.05567

IF: 5.414

2022-10-27

Machine Learning

Abstract:Many problems in science and engineering can be represented by a set of partial differential equations (PDEs) through mathematical modeling. Mechanism-based computation following PDEs has long been an essential paradigm for studying topics such as computational fluid dynamics, multiphysics simulation, molecular dynamics, or even dynamical systems. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. At the same time, solving PDEs efficiently has been a long-standing challenge. Generally, except for a few differential equations for which analytical solutions are directly available, many more equations must rely on numerical approaches such as the finite difference method, finite element method, finite volume method, and boundary element method to be solved approximately. These numerical methods usually divide a continuous problem domain into discrete points and then concentrate on solving the system at each of those points. Though the effectiveness of these traditional numerical methods, the vast number of iterative operations accompanying each step forward significantly reduces the efficiency. Recently, another equally important paradigm, data-based computation represented by deep learning, has emerged as an effective means of solving PDEs. Surprisingly, a comprehensive review for this interesting subfield is still lacking. This survey aims to categorize and review the current progress on Deep Neural Networks (DNNs) for PDEs. We discuss the literature published in this subfield over the past decades and present them in a common taxonomy, followed by an overview and classification of applications of these related methods in scientific research and engineering scenarios. The origin, developing history, character, sort, as well as the future trends in each potential direction of this subfield are also introduced.
Deep learning methods for partial differential equations and related parameter identification problems

Derick Nganyu Tanyu,Jianfeng Ning,Tom Freudenberg,Nick Heilenkoetter,Andreas Rademacher,Uwe Iben,Peter Maass

DOI: https://doi.org/10.1088/1361-6420/ace9d4

IF: 2.408

2023-07-26

Inverse Problems

Abstract:Recent years have witnessed a growth in mathematics for deep learning—which seeks a deeper understanding of the concepts of deep learning with mathematics and explores how to make it more robust—and deep learning for mathematics, where deep learning algorithms are used to solve problems in mathematics. The latter has popularised the field of scientific machine learning where deep learning is applied to problems in scientific computing. Specifically, more and more neural network architectures have been developed to solve specific classes of partial differential equations (PDEs). Such methods exploit properties that are inherent to PDEs and thus solve the PDEs better than standard feed-forward neural networks, recurrent neural networks, or convolutional neural networks. This has had a great impact in the area of mathematical modeling where parametric PDEs are widely used to model most natural and physical processes arising in science and engineering. In this work, we review such methods as well as their extensions for parametric studies and for solving the related inverse problems. We equally proceed to show their relevance in some industrial applications.

mathematics, applied,physics, mathematical
DL-PDE: Deep-learning based data-driven discovery of partial differential equations from discrete and noisy data

Hao Xu,Haibin Chang,Dongxiao Zhang

DOI: https://doi.org/10.4208/cicp.OA-2020-0142

2020-04-06

Abstract:In recent years, data-driven methods have been developed to learn dynamical systems and partial differential equations (PDE). The goal of such work is discovering unknown physics and the corresponding equations. However, prior to achieving this goal, major challenges remain to be resolved, including learning PDE under noisy data and limited discrete data. To overcome these challenges, in this work, a deep-learning based data-driven method, called DL-PDE, is developed to discover the governing PDEs of underlying physical processes. The DL-PDE method combines deep learning via neural networks and data-driven discovery of PDE via sparse regressions. In the DL-PDE, a neural network is first trained, and then a large amount of meta-data is generated, and the required derivatives are calculated by automatic differentiation. Finally, the form of PDE is discovered by sparse regression. The proposed method is tested with physical processes, governed by groundwater flow equation, convection-diffusion equation, Burgers equation and Korteweg-de Vries (KdV) equation, for proof-of-concept and applications in real-world engineering settings. The proposed method achieves satisfactory results when data are noisy and limited.

Machine Learning,Computational Physics
Solving High-Dimensional Partial Differential Equations Using Deep Learning.

Jiequn Han,Arnulf Jentzen,Weinan E

DOI: https://doi.org/10.1073/pnas.1718942115

2018-01-01

Abstract:Developing algorithms for solving high-dimensional partial differential equations (PDEs) has been an exceedingly difficult task for a long time, due to the notoriously difficult problem known as the "curse of dimensionality." This paper introduces a deep learning-based approach that can handle general high-dimensional parabolic PDEs. To this end, the PDEs are reformulated using backward stochastic differential equations and the gradient of the unknown solution is approximated by neural networks, very much in the spirit of deep reinforcement learning with the gradient acting as the policy function. Numerical results on examples including the nonlinear Black-Scholes equation, the Hamilton-Jacobi-Bellman equation, and the Allen-Cahn equation suggest that the proposed algorithm is quite effective in high dimensions, in terms of both accuracy and cost. This opens up possibilities in economics, finance, operational research, and physics, by considering all participating agents, assets, resources, or particles together at the same time, instead of making ad hoc assumptions on their interrelationships.
Deep-learning based discovery of partial differential equations in integral form from sparse and noisy data

Hao Xu,Dongxiao Zhang,Nanzhe Wang

DOI: https://doi.org/10.1016/j.jcp.2021.110592

IF: 4.645

2021-11-01

Journal of Computational Physics

Abstract:Data-driven discovery of partial differential equations (PDEs) has attracted increasing attention in recent years. Although significant progress has been made, certain unresolved issues remain. For example, for PDEs with high-order derivatives, the performance of existing methods is unsatisfactory, especially when the data are sparse and noisy. It is also difficult to discover heterogeneous parametric PDEs where heterogeneous parameters are embedded in the partial differential operators. In this work, a new framework combining deep-learning and integral form is proposed to handle the above-mentioned problems simultaneously, and improve the accuracy and stability of PDE discovery. In the framework, a deep neural network is firstly trained with observation data to generate meta-data and calculate derivatives. Then, a unified integral form is defined, and the genetic algorithm is employed to discover the best structure. Finally, the values of parameters are calculated, and whether the parameters are constants or variables is identified. Numerical experiments proved that our proposed algorithm is more robust to noise and more accurate compared with existing methods due to the utilization of integral form. Our proposed algorithm is also able to discover PDEs with high-order derivatives or heterogeneous parameters accurately with sparse and noisy data.

physics, mathematical,computer science, interdisciplinary applications
Surrogate-Based Physics-Informed Neural Networks for Elliptic Partial Differential Equations

Peng Zhi,Yuching Wu,Cheng Qi,Tao Zhu,Xiao Wu,Hongyu Wu

DOI: https://doi.org/10.3390/math11122723

IF: 2.4

2023-06-17

Mathematics

Abstract:The purpose of this study is to investigate the role that a deep learning approach could play in computational mechanics. In this paper, a convolutional neural network technique based on modified loss function is proposed as a surrogate of the finite element method (FEM). Several surrogate-based physics-informed neural networks (PINNs) are developed to solve representative boundary value problems based on elliptic partial differential equations (PDEs). According to the authors' knowledge, the proposed method has been applied for the first time to solve boundary value problems with elliptic partial differential equations as the governing equations. The results of the proposed surrogate-based approach are in good agreement with those of the conventional FEM. It is found that modification of the loss function could improve the prediction accuracy of the neural network. It is demonstrated that to some extent, the deep learning approach could replace the conventional numerical method as a significant surrogate model.

mathematics
An Efficient Method for Solving Two-Dimensional Partial Differential Equations with the Deep Operator Network

Xiaoyu Zhang,Yichao Wang,Xiting Peng,Chaofeng Zhang

DOI: https://doi.org/10.3390/axioms12121095

2023-11-29

Axioms

Abstract:Partial differential equations (PDEs) usually apply for modeling complex physical phenomena in the real world, and the corresponding solution is the key to interpreting these problems. Generally, traditional solving methods suffer from inefficiency and time consumption. At the same time, the current rise in machine learning algorithms, represented by the Deep Operator Network (DeepONet), could compensate for these shortcomings and effectively predict the solutions of PDEs by learning the operators from the data. The current deep learning-based methods focus on solving one-dimensional PDEs, but the research on higher-dimensional problems is still in development. Therefore, this paper proposes an efficient scheme to predict the solution of two-dimensional PDEs with improved DeepONet. In order to construct the data needed for training, the functions are sampled from a classical function space and produce the corresponding two-dimensional data. The difference method is used to obtain the numerical solutions of the PDEs and form a point-value data set. For training the network, the matrix representing two-dimensional functions is processed to form vectors and adapt the DeepONet model perfectly. In addition, we theoretically prove that the discrete point division of the data ensures that the model loss is guaranteed to be in a small range. This method is verified for predicting the two-dimensional Poisson equation and heat conduction equation solutions through experiments. Compared with other methods, the proposed scheme is simple and effective.
Solving Irregular and Data-enriched Differential Equations using Deep Neural Networks

Craig Michoski,Milos Milosavljevic,Todd Oliver,David Hatch

DOI: https://doi.org/10.48550/arXiv.1905.04351

2019-05-11

Abstract:Recent work has introduced a simple numerical method for solving partial differential equations (PDEs) with deep neural networks (DNNs). This paper reviews and extends the method while applying it to analyze one of the most fundamental features in numerical PDEs and nonlinear analysis: irregular solutions. First, the Sod shock tube solution to compressible Euler equations is discussed, analyzed, and then compared to conventional finite element and finite volume methods. These methods are extended to consider performance improvements and simultaneous parameter space exploration. Next, a shock solution to compressible magnetohydrodynamics (MHD) is solved for, and used in a scenario where experimental data is utilized to enhance a PDE system that is \emph{a priori} insufficient to validate against the observed/experimental data. This is accomplished by enriching the model PDE system with source terms and using supervised training on synthetic experimental data. The resulting DNN framework for PDEs seems to demonstrate almost fantastical ease of system prototyping, natural integration of large data sets (be they synthetic or experimental), all while simultaneously enabling single-pass exploration of the entire parameter space.

Machine Learning,Numerical Analysis,Computational Physics,Data Analysis, Statistics and Probability
A Survey on Solving and Discovering Differential Equations Using Deep Neural Networks

Hyeonjung,Jung,Jayant Gupta,Bharat Jayaprakash,Matthew Eagon,Harish Panneer Selvam,Carl Molnar,William Northrop,Shashi Shekhar

2023-06-19

Abstract:Ordinary and partial differential equations (DE) are used extensively in scientific and mathematical domains to model physical systems. Current literature has focused primarily on deep neural network (DNN) based methods for solving a specific DE or a family of DEs. Research communities with a history of using DE models may view DNN-based differential equation solvers (DNN-DEs) as a faster and transferable alternative to current numerical methods. However, there is a lack of systematic surveys detailing the use of DNN-DE methods across physical application domains and a generalized taxonomy to guide future research. This paper surveys and classifies previous works and provides an educational tutorial for senior practitioners, professionals, and graduate students in engineering and computer science. First, we propose a taxonomy to navigate domains of DE systems studied under the umbrella of DNN-DE. Second, we examine the theory and performance of the Physics Informed Neural Network (PINN) to demonstrate how the influential DNN-DE architecture mathematically solves a system of equations. Third, to reinforce the key ideas of solving and discovery of DEs using DNN, we provide a tutorial using DeepXDE, a Python package for developing PINNs, to develop DNN-DEs for solving and discovering a classic DE, the linear transport equation.

Neural and Evolutionary Computing
Improved Deep Neural Networks with Domain Decomposition in Solving Partial Differential Equations

Wu Wei,Feng Xinlong,Xu Hui

DOI: https://doi.org/10.1007/s10915-022-01980-y

2022-01-01

Journal of Scientific Computing

Abstract:An improved neural networks method based on domain decomposition is proposed to solve partial differential equations, which is an extension of the physics informed neural networks (PINNs). Although recent research has shown that PINNs perform effectively in solving partial differential equations, they still have difficulties in solving large-scale complex problems, due to using a single neural network and gradient pathology. In this paper, the proposed approach aims at implementing calculations on sub-domains and improving the expressiveness of neural networks to mitigate gradient pathology. By investigations, it is shown that, although the neural networks structure and the loss function are complicated, the proposed method outperforms the classical PINNs with respect to training effectiveness, computational accuracy, and computational cost.
Study on a 3D Possion's Equation Slover Based on Deep Learning Technique

Tao Shan,Xunwang Dang,Maokun Li,Fan Yang,Shenheng Xu,Ji Wu

DOI: https://doi.org/10.1109/COMPEM.2018.8496657

2018-01-01

Abstract:In this study, we investigate the feasibility of applying deep learning technique to build a 3D electrostatic solver. A deep convolutional neural network (CNN) is proposed to take advantage of the power of CNN in approximation of highly nonlinear functions and prediction of the potential distribution of electrostatic field. Compared with traditional numerical solvers based on finite difference scheme, this method uses a data-driven end-to-end model. Numerical experiments show that the prediction error can reach below 3 percent and the computing time can be significantly reduced compared with traditional finite difference solvers.
Symbolically Solving Partial Differential Equations using Deep Learning

Maysum Panju,Kourosh Parand,Ali Ghodsi

DOI: https://doi.org/10.48550/arXiv.2011.06673

IF: 5.414

2020-11-12

Machine Learning

Abstract:We describe a neural-based method for generating exact or approximate solutions to differential equations in the form of mathematical expressions. Unlike other neural methods, our system returns symbolic expressions that can be interpreted directly. Our method uses a neural architecture for learning mathematical expressions to optimize a customizable objective, and is scalable, compact, and easily adaptable for a variety of tasks and configurations. The system has been shown to effectively find exact or approximate symbolic solutions to various differential equations with applications in natural sciences. In this work, we highlight how our method applies to partial differential equations over multiple variables and more complex boundary and initial value conditions.
Deep-learning of parametric partial differential equations from sparse and noisy data

Hao Xu,Dongxiao Zhang,Junsheng Zeng

DOI: https://doi.org/10.1063/5.0042868

IF: 4.6

2021-03-01

Physics of Fluids

Abstract:Data-driven methods have recently made great progress in the discovery of partial differential equations (PDEs) from spatial-temporal data. However, several challenges remain to be solved, including sparse noisy data, incomplete library, and spatially or temporally varying coefficients. In this work, a new framework, which combines neural network, genetic algorithm, and stepwise methods, is put forward to address all of these challenges simultaneously. In the framework, a trained neural network is utilized to calculate derivatives and generate a large amount of meta-data, which solves the problem of sparse noisy data. Next, the genetic algorithm is used to discover the form of PDEs and corresponding coefficients, which solves the problem of the incomplete initial library. Finally, a stepwise adjustment method is introduced to discover parametric PDEs with spatially or temporally varying coefficients. In this method, the structure of a parametric PDE is first discovered, and then the general form of varying coefficients is identified. The proposed algorithm is tested on the Burgers equation, the convection-diffusion equation, the wave equation, and the KdV equation. Results demonstrate that this method is robust to sparse and noisy data, and is able to discover parametric PDEs with an incomplete initial library.

mechanics,physics, fluids & plasmas
Convergence Analysis of a Quasi-Monte Carlo-based Deep Learning Algorithm for Solving Partial Differential Equations

Fengjiang Fu,Xiaoqun Wang

DOI: https://doi.org/10.4208/nmtma.oa-2022-0166

2023-01-01

Numerical Mathematics Theory Methods and Applications

Abstract:Deep learning methods have achieved great success in solving partial differential equations (PDEs), where the loss is often defined as an integral. The accuracy and efficiency of these algorithms depend greatly on the quadrature method. We propose to apply quasi-Monte Carlo (QMC) methods to the Deep Ritz Method (DRM) for solving the Neumann problems for the Poisson equation and the static Schrödinger equation. For error estimation, we decompose the error of using the deep learning algorithm to solve PDEs into the generalization error, the approximation error and the training error. We establish the upper bounds and prove that QMC-based DRM achieves an asymptotically smaller error bound than DRM. Numerical experiments show that the proposed method converges faster in all cases and the variances of the gradient estimators of randomized QMC-based DRM are much smaller than those of DRM, which illustrates the superiority of QMC in deep learning over MC.
A Non-Gradient Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks.

Yifan Peng,Dan Hu,Zin-Qin John Xu

DOI: https://doi.org/10.1016/j.jcp.2022.111690

2022-01-01

SSRN Electronic Journal

Abstract:Deep learning has achieved wide success in solving Partial Differential Equations (PDEs), with particular strength in handling high dimensional problems and parametric problems. Nevertheless, there is still a lack of a clear picture on the designing of network architecture and the training of network parameters. In this work, we developed a non-gradient framework for solving elliptic PDEs based on Neural Tangent Kernel (NTK): 1. ReLU activation function is used to control the compactness of the NTK so that solutions with relatively high frequency components can be well expressed; 2. Numerical discretization is used for differential operators to reduce computational cost; 3. A dissipative evolution dynamics corresponding to the elliptic PDE is used for parameter training instead of the gradient-type descent of a loss function. The dissipative dynamics can guarantee the convergence of the training process while avoiding employment of loss functions with high order derivatives. It is also helpful for both controlling of kernel property and reduction of computational cost. Numerical tests have shown excellent performance of the non-gradient method.
Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networks

Xiang Huang,Hongsheng Liu,Beiji Shi,Zidong Wang,Kang Yang,Yang Li,Bingya Weng,Min Wang,Haotian Chu,Jing Zhou,Fan Yu,Bei Hua,Lei Chen,Bin Dong

2021-11-02

Abstract:In recent years, deep learning technology has been used to solve partial differential equations (PDEs), among which the physics-informed neural networks (PINNs) emerges to be a promising method for solving both forward and inverse PDE problems. PDEs with a point source that is expressed as a Dirac delta function in the governing equations are mathematical models of many physical processes. However, they cannot be solved directly by conventional PINNs method due to the singularity brought by the Dirac delta function. We propose a universal solution to tackle this problem with three novel techniques. Firstly the Dirac delta function is modeled as a continuous probability density function to eliminate the singularity; secondly a lower bound constrained uncertainty weighting algorithm is proposed to balance the PINNs losses between point source area and other areas; and thirdly a multi-scale deep neural network with periodic activation function is used to improve the accuracy and convergence speed of the PINNs method. We evaluate the proposed method with three representative PDEs, and the experimental results show that our method outperforms existing deep learning-based methods with respect to the accuracy, the efficiency and the versatility.

Artificial Intelligence,Computational Physics
An Overview on Machine Learning Methods for Partial Differential Equations: from Physics Informed Neural Networks to Deep Operator Learning

Lukas Gonon,Arnulf Jentzen,Benno Kuckuck,Siyu Liang,Adrian Riekert,Philippe von Wurstemberger

2024-08-24

Abstract:The approximation of solutions of partial differential equations (PDEs) with numerical algorithms is a central topic in applied mathematics. For many decades, various types of methods for this purpose have been developed and extensively studied. One class of methods which has received a lot of attention in recent years are machine learning-based methods, which typically involve the training of artificial neural networks (ANNs) by means of stochastic gradient descent type optimization methods. While approximation methods for PDEs using ANNs have first been proposed in the 1990s they have only gained wide popularity in the last decade with the rise of deep learning. This article aims to provide an introduction to some of these methods and the mathematical theory on which they are based. We discuss methods such as physics-informed neural networks (PINNs) and deep BSDE methods and consider several operator learning approaches.

Numerical Analysis,Machine Learning
D3M: A deep domain decomposition method for partial differential equations

Ke Li,Kejun Tang,Tianfan Wu,Qifeng Liao

DOI: https://doi.org/10.1109/ACCESS.2019.2957200

2019-09-24

Abstract:A state-of-the-art deep domain decomposition method (D3M) based on the variational principle is proposed for partial differential equations (PDEs). The solution of PDEs can be formulated as the solution of a constrained optimization problem, and we design a multi-fidelity neural network framework to solve this optimization problem. Our contribution is to develop a systematical computational procedure for the underlying problem in parallel with domain decomposition. Our analysis shows that the D3M approximation solution converges to the exact solution of underlying PDEs. Our proposed framework establishes a foundation to use variational deep learning in large-scale engineering problems and designs. We present a general mathematical framework of D3M, validate its accuracy and demonstrate its efficiency with numerical experiments.

Machine Learning,Numerical Analysis

Some Investigations on Applying Deep Learning Techniques to Solve Partial Differential Equations

Solving Partial Differential Equations Using Deep Learning and Physical Constraints

Study on a Poisson's Equation Solver Based on Deep Learning Technique.

Partial Differential Equations Meet Deep Neural Networks: A Survey

Deep learning methods for partial differential equations and related parameter identification problems

DL-PDE: Deep-learning based data-driven discovery of partial differential equations from discrete and noisy data

Solving High-Dimensional Partial Differential Equations Using Deep Learning.

Deep-learning based discovery of partial differential equations in integral form from sparse and noisy data

Surrogate-Based Physics-Informed Neural Networks for Elliptic Partial Differential Equations

An Efficient Method for Solving Two-Dimensional Partial Differential Equations with the Deep Operator Network

Solving Irregular and Data-enriched Differential Equations using Deep Neural Networks

A Survey on Solving and Discovering Differential Equations Using Deep Neural Networks

Improved Deep Neural Networks with Domain Decomposition in Solving Partial Differential Equations

Study on a 3D Possion's Equation Slover Based on Deep Learning Technique

Symbolically Solving Partial Differential Equations using Deep Learning

Deep-learning of parametric partial differential equations from sparse and noisy data

Convergence Analysis of a Quasi-Monte Carlo-based Deep Learning Algorithm for Solving Partial Differential Equations

A Non-Gradient Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks.

Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networks

An Overview on Machine Learning Methods for Partial Differential Equations: from Physics Informed Neural Networks to Deep Operator Learning

D3M: A deep domain decomposition method for partial differential equations