Abstract:Physics-informed deep learning has emerged as a promising alternative for solving partial differential equations. However, for complex problems, training these networks can still be challenging, often resulting in unsatisfactory accuracy and efficiency. In this work, we demonstrate that the failure of plain physics-informed neural networks arises from the significant discrepancy in the convergence speed of residuals at different training points, where the slowest convergence speed dominates the overall solution convergence. Based on these observations, we propose a point-wise adaptive weighting method that balances the residual decay rate across different training points. The performance of our proposed adaptive weighting method is compared with current state-of-the-art adaptive weighting methods on benchmark problems for both physics-informed neural networks and physics-informed deep operator networks. Through extensive numerical results we demonstrate that our proposed approach of balanced residual decay rates offers several advantages, including bounded weights, high prediction accuracy, fast convergence speed, low training uncertainty, low computational cost and ease of hyperparameter tuning.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the convergence and accuracy problems encountered by Physics - Informed Neural Networks (PINNs) and Physics - Informed Deep Operator Networks (PIDeopONets) when solving partial differential equations (PDEs). Specifically, the author observes: 1. **Significant differences in the residual convergence rates at different training points**: In complex physical problems, the residual convergence rates (i.e., the errors of PDEs and boundary conditions) at different training points differ by several orders of magnitude. 2. **The slowest convergence rate dominates the convergence of the overall solution**: Since the slowest convergence rate determines the convergence of the overall solution, this leads to a decline in overall performance during the training process. Based on these observations, the author proposes a new adaptive weighting method to improve the performance of PINNs and PIDeopONets by balancing the residual decay rates at different training points. The main objectives of this method are: - **Improve prediction accuracy**: By more evenly distributing the weights, ensure that all physical laws can be learned to the same extent by the model. - **Accelerate the convergence rate**: Reduce the uncertainty in training, lower the computational cost, and simplify hyper - parameter adjustment. ### Method overview The author introduces the concept of "Inverse Residual Decay Rate" (IRDR), which is defined as: \[ \text{irdr} = \frac{R^2(t)}{\sqrt{\hat{R}^4(t)+\epsilon}} \] where \( R(t) \) is the residual, \( \hat{R}^4(t) \) is the exponential moving average of the squared residual, and \(\epsilon\) is a very small positive number to avoid division - by - zero errors. To balance the residual decay rates, the author proposes the following steps: 1. **Calculate the IRDR for each training point**: Calculate the IRDR for each training point according to the current iteration step. 2. **Normalize the weights**: Set the weight for each training point to the normalized value of the IRDR. 3. **Update the weights**: Update the weights using the exponential moving average method to filter out the noise during the training process. 4. **Maximize the learning rate**: Maximize the learning rate by adjusting the scaling factor \( s \) to accelerate the training process. Through these improvements, the author demonstrates the superior performance of this method on multiple benchmark problems, including higher prediction accuracy, faster convergence rate, and lower training uncertainty. ### Conclusion The adaptive weighting method based on balancing the residual decay rate proposed in this paper significantly improves the performance of PINNs and PIDeopONets in solving complex physical problems and solves the convergence and accuracy problems existing in traditional methods.

Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks

Stable Weight Updating: A Key to Reliable PDE Solutions Using Deep Learning

Self-adaptive loss balanced Physics-informed neural networks

Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks

Residual-based attention in physics-informed neural networks

Optimally weighted loss functions for solving PDEs with Neural Networks

Power-Enhanced Residual Network for Function Approximation and Physics-Informed Inverse Problems

Enhanced physics‐informed neural networks for hyperelasticity

Self-adaptive loss balanced Physics-informed neural networks for the incompressible Navier-Stokes equations

Dynamic Weight Strategy of Physics-Informed Neural Networks for the 2D Navier–Stokes Equations

Physics-informed neural networks based on adaptive weighted loss functions for Hamilton-Jacobi equations

Parallel Physics-Informed Neural Networks with Bidirectional Balance

Importance of hyper-parameter optimization during training of physics-informed deep learning networks

Simple yet effective adaptive activation functions for physics-informed neural networks

Enhancing Convergence Speed with Feature-Enforcing Physics-Informed Neural Networks: Utilizing Boundary Conditions as Prior Knowledge for Faster Convergence

Multi-Objective Loss Balancing for Physics-Informed Deep Learning

Loss-attentional physics-informed neural networks

Adaptive Task Decomposition Physics-Informed Neural Networks

Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks