Abstract:This study investigates the impact of Sobolev Training on operator learning frameworks for improving model performance. Our research reveals that integrating derivative information into the loss function enhances the training process, and we propose a novel framework to approximate derivatives on irregular meshes in operator learning. Our findings are supported by both experimental evidence and theoretical analysis. This demonstrates the effectiveness of Sobolev Training in approximating the solution operators between infinite-dimensional spaces.

What problem does this paper attempt to address?

This paper mainly discusses the application of Sobolev training methods to the operation learning framework to improve model performance. Sobolev training optimizes the loss function by integrating the derivative information of the objective function, thereby improving the training process. A new algorithm is proposed in this study to approximate derivatives on irregular grids, especially when dealing with complex partial differential equations in physical modeling. Through experimental and theoretical analysis, the effectiveness of Sobolev training in approximating solution operators in infinite-dimensional space is demonstrated. The paper first introduces the progress of machine learning in fields such as computer vision, natural language processing, and physical modeling, with a special mention of the FNO and DeepONet models in the operation learning field. The concept of Sobolev training was initially introduced in the work of Czarnecki et al. (2017), which improves prediction accuracy, data efficiency, and model generalization ability. Sobolev training has also been applied to Physics-Informed Neural Networks (PINNs) to accelerate the convergence speed of solving complex PDEs. The main contributions of the paper include: 1. Proposing an algorithm to approximate derivatives on irregular grids in order to apply Sobolev training to existing operation learning models. 2. Conducting convergence analysis for the first time in the operation learning field. 3. Introducing Sobolev training to operation learning and providing theoretical and empirical support. In addition, the paper discusses how to use PCGrad to optimize Sobolev training to improve model performance, and demonstrates the effectiveness of Sobolev training on different models and datasets through experiments, proving its significant improvement in model performance. The experimental results show that Sobolev training has good robustness when facing noise, and for certain tasks, the error rate is reduced by more than 30%.

Sobolev Training for Operator Learning

Solving PDE-constrained Control Problems Using Operator Learning

DeepONet for Solving PDEs: Generalization Analysis in Sobolev Training

Improved architectures and training algorithms for deep operator networks

An Operator Learning Approach to Nonsmooth Optimal Control of Nonlinear PDEs

Variational operator learning: A unified paradigm marrying training neural operators and solving partial differential equations

Variational operator learning: A unified paradigm for training neural operators and solving partial differential equations

Operator Learning: Algorithms and Analysis

Minimax Optimal Kernel Operator Learning via Multilevel Training

Transfer Operator Learning with Fusion Frame

An approximate operator-based learning method for the numerical solutions of stochastic differential equations

Learning Operators with Stochastic Gradient Descent in General Hilbert Spaces

Derivative-Informed Neural Operator: An Efficient Framework for High-Dimensional Parametric Derivative Learning

Optimal deep learning of holomorphic operators between Banach spaces

Diffeomorphic Latent Neural Operators for Data-Efficient Learning of Solutions to Partial Differential Equations

An Operator Learning Framework for Spatiotemporal Super-resolution of Scientific Simulations

Operator learning based on sparse high-dimensional approximation

Operator SVD with Neural Networks via Nested Low-Rank Approximation

Solving Partial Differential Equations in Different Domains by Operator Learning method Based on Boundary Integral Equations

Learning nonlocal regularization operators