Abstract:In the recent years, Physics Informed Neural Networks (PINNs) have received strong interest as a method to solve PDE driven systems, in particular for data assimilation purpose. This method is still in its infancy, with many shortcomings and failures that remain not properly understood. In this paper we propose a natural gradient approach to PINNs which contributes to speed-up and improve the accuracy of the training. Based on an in depth analysis of the differential geometric structures of the problem, we come up with two distinct contributions: (i) a new natural gradient algorithm that scales as $\min(P^2S, S^2P)$, where $P$ is the number of parameters, and $S$ the batch size; (ii) a mathematically principled reformulation of the PINNs problem that allows the extension of natural gradient to it, with proved connections to Green's function theory.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problems of training efficiency and accuracy encountered by Physics Informed Neural Networks (PINNs) when solving partial differential equations (PDEs). Specifically, the author proposes a new natural gradient method (ANaGRAM) to accelerate the training of PINNs and improve their accuracy. #### Main problems and challenges 1. **Low training efficiency**: The existing PINNs methods have the problem of slow convergence during the training process. 2. **Insufficient accuracy**: In the absence of additional data, the existing methods are difficult to obtain high - accuracy approximate solutions. 3. **Insufficient theoretical understanding**: There is a lack of in - depth understanding of the geometric structure and optimization dynamics during the PINNs training process. #### Solutions To solve the above problems, the author makes the following two main contributions: 1. **New natural gradient algorithm**: - A new natural gradient algorithm (ANaGRAM) with a complexity of $O(\min(P^{2}S, S^{2}P))$ is proposed, where $P$ is the number of parameters and $S$ is the batch size. This is a reduction compared to the traditional $O(P^{3})$ complexity. - By introducing the empirical tangent space and the empirical natural gradient, the training process of PINNs is improved. 2. **Re - formulation in mathematical principles**: - The PINNs framework is re - interpreted from the perspective of functional analysis, so that the natural gradient method can be applied to PINNs. - A direct correspondence between the natural gradient and the Green's function is established, revealing that the natural gradient update implicitly uses the Green's function of the operator. #### Experimental verification The author verifies the effectiveness of ANaGRAM through multiple experiments, including the two - dimensional Laplace equation, the one - dimensional heat conduction equation, the five - dimensional Laplace equation and the one - dimensional Allen - Cahn equation. The experimental results show that ANaGRAM can significantly improve the solution accuracy while maintaining a low computational cost, and performs well on nonlinear PDEs. #### Conclusion In conclusion, this paper provides an efficient and accurate PINNs training method by introducing ANaGRAM, which solves the deficiencies of existing methods in training efficiency and accuracy. Future research directions include further optimizing the batch point selection strategy and hyper - parameter adjustment methods.

ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning

Achieving High Accuracy with PINNs via Energy Natural Gradients

Enhancing PINNs for solving PDEs via adaptive collocation point movement and adaptive loss weighting

Improved physics-informed neural network in mitigating gradient related failures

Gradient-Annihilated PINNs for Solving Riemann Problems: Application to Relativistic Hydrodynamics

Learning Physics-Informed Neural Networks without Stacked Back-propagation

Generalized conditional symmetry enhanced physics-informed neural network and application to the forward and inverse problems of nonlinear diffusion equations

Revisiting PINNs: Generative Adversarial Physics-informed Neural Networks and Point-weighting Method

Residual-based attention and connection to information bottleneck theory in PINNs

Physics-Informed Neural Networks (PINNs) for Parameterized PDEs: A Metalearning Approach

Physics-informed neural network based on a new adaptive gradient descent algorithm for solving partial differential equations of flow problems

A metalearning approach for Physics-Informed Neural Networks (PINNs): Application to parameterized PDEs

Annealed adaptive importance sampling method in PINNs for solving high dimensional partial differential equations

Adaptive Self-supervision Algorithms for Physics-informed Neural Networks

CP-PINNs: Data-Driven Changepoints Detection in PDEs Using Online Optimized Physics-Informed Neural Networks

DiffGrad for Physics-Informed Neural Networks

A Gradient-Enhanced Physics-Informed Neural Networks Method for the Wave Equation

iPINNs: incremental learning for Physics-informed neural networks

Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamics

Accelerated Training of Physics-Informed Neural Networks (PINNs) using Meshless Discretizations

Dual Cone Gradient Descent for Training Physics-Informed Neural Networks