Abstract:Physics-informed <a class="topic-link" href="/topics/computer-science/neural-networks">neural networks</a> (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy <a class="topic-link" href="/topics/computer-science/observational-data">observational data</a>, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random <a class="topic-link" href="/topics/engineering/fourier-feature">Fourier features</a>, and justify how such coordinate embedding layers can lead to robust and accurate <a class="topic-link" href="/topics/computer-science/neural-network-model">PINN models</a>. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction–diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and <a class="topic-link" href="/topics/computer-science/inverse-problem">inverse problems</a> involving <a class="topic-link" href="/topics/engineering/partial-differential-equation">partial differential equations</a> with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at <a href="https://github.com/PredictiveIntelligenceLab/MultiscalePINNs">https://github.com/PredictiveIntelligenceLab/MultiscalePINNs</a>.

On the spectral bias of coupled frequency predictor–corrector triangular DNN: The convergence analysis

Learning high frequency data via the coupled frequency predictor-corrector triangular DNN

On the exact computation of linear frequency principle dynamics and its generalization

Understanding the dynamics of the frequency bias in neural networks

The Spectral Bias of Polynomial Neural Networks

Frequency Bias in Neural Networks for Input of Non-Uniform Density

Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks

A Fine-Grained Spectral Perspective on Neural Networks

Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limit

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Analyzing Convergence in Quantum Neural Networks: Deviations from Neural Tangent Kernels

Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks

A neuron-wise subspace correction method for the finite neuron method

Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

When and why PINNs fail to train: A neural tangent kernel perspective

Overview frequency principle/spectral bias in deep learning

On Exact Computation with an Infinitely Wide Neural Net

A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks

FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Convergence Analysis