Abstract:Modeling and simulation of complex non-linear systems are essential in physics, engineering, and signal processing. Neural networks are widely regarded for such tasks due to their ability to learn complex representations from data. Training deep neural networks traditionally requires large amounts of data, which may not always be readily available for such systems. Contrarily, there is a large amount of domain knowledge in the form of mathematical models for the physics/behavior of such systems. A new class of neural networks called Physics-Informed Neural Networks (PINNs) has gained much attention recently as a paradigm for combining physics into neural networks. They have become a powerful tool for solving forward and inverse problems involving differential equations. A general framework of a PINN consists of a multi-layer perceptron that learns the solution of the partial differential equation (PDE) along with its boundary/initial conditions by minimizing a multi-objective loss function. This is formed by the sum of individual loss terms that penalize the output at different collocation points based on the differential equation and initial and boundary conditions. However, multiple loss terms arising from PDE residual and boundary conditions in PINNs pose a challenge in optimizing the overall loss function. This often leads to training failures and inaccurate results. We propose advanced gradient statistics-based weighting schemes for PINNs to address this challenge. These schemes utilize backpropagated gradient statistics of individual loss terms to appropriately scale and assign weights to each term, ensuring balanced training and meaningful solutions. In addition to the existing gradient statistics-based weighting schemes, we introduce kurtosis-standard deviation-based and combined mean and standard deviation-based schemes for approximating solutions of PDEs using PINNs. We provide a qualitative and quantitative comparison of these weighting schemes on 2D Poisson's and Klein-Gordon's equations, highlighting their effectiveness in improving PINN performance.

Gradient-optimized physics-informed neural networks (GOPINNs): a deep learning method for solving the complex modified KdV equation

A Gradient-Enhanced Physics-Informed Neural Networks Method for the Wave Equation

An Improved Physics-Informed Neural Network Based on a New Adaptive Gradient Descent Algorithm for Solving Partial Differential Equations of Nonlinear Systems

Improved physics-informed neural network in mitigating gradient related failures

Gradient Statistics-Based Multi-Objective Optimization in Physics-Informed Neural Networks

Data-Guided Physics-Informed Neural Networks for Solving Inverse Problems in Partial Differential Equations

Physics-informed neural network based on a new adaptive gradient descent algorithm for solving partial differential equations of flow problems

RPINNs: Rectified-physics informed neural networks for solving stationary partial differential equations

Advanced Physics-Informed Neural Networks for Numerical Approximation of the Coupled Schrödinger-Kdv Equation

A Second-Order Network Structure Based on Gradient-Enhanced Physics-Informed Neural Networks for Solving Parabolic Partial Differential Equations.

Gradient auxiliary physics-informed neural network for nonlinear biharmonic equation

Gradient-enhanced physics-informed neural networks based on transfer learning for inverse problems of the variable coefficient differential equations

Gradient-Annihilated PINNs for Solving Riemann Problems: Application to Relativistic Hydrodynamics

Data-driven solutions and parameter estimations of a family of higher-order KdV equations based on physics informed neural networks

Dual Cone Gradient Descent for Training Physics-Informed Neural Networks

DiffGrad for Physics-Informed Neural Networks

Binary structured physics-informed neural networks for solving equations with rapidly changing solutions

Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks

Parallel Physics-Informed Neural Networks Method with Regularization Strategies for the Forward-Inverse Problems of the Variable Coefficient Modified KdV Equation

Physics-informed neural networks based on adaptive weighted loss functions for Hamilton-Jacobi equations