Kolmogorov n-Widths for Multitask Physics-Informed Machine Learning (PIML) Methods: Towards Robust Metrics

Michael Penwarden,Houman Owhadi,Robert M. Kirby

2024-09-05

Abstract:Physics-informed machine learning (PIML) as a means of solving partial differential equations (PDE) has garnered much attention in the Computational Science and Engineering (CS&E) world. This topic encompasses a broad array of methods and models aimed at solving a single or a collection of PDE problems, called multitask learning. PIML is characterized by the incorporation of physical laws into the training process of machine learning models in lieu of large data when solving PDE problems. Despite the overall success of this collection of methods, it remains incredibly difficult to analyze, benchmark, and generally compare one approach to another. Using Kolmogorov n-widths as a measure of effectiveness of approximating functions, we judiciously apply this metric in the comparison of various multitask PIML architectures. We compute lower accuracy bounds and analyze the model's learned basis functions on various PDE problems. This is the first objective metric for comparing multitask PIML architectures and helps remove uncertainty in model validation from selective sampling and overfitting. We also identify avenues of improvement for model architectures, such as the choice of activation function, which can drastically affect model generalization to "worst-case" scenarios, which is not observed when reporting task-specific errors. We also incorporate this metric into the optimization process through regularization, which improves the models' generalizability over the multitask PDE problem.

Machine Learning,Computational Physics

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily focuses on some key issues faced by Physics-Informed Machine Learning (PIML) methods when solving Partial Differential Equations (PDEs). Specifically:

1. **Comparison and Evaluation of Different Architectures**:
- Although there are many different PIML methods (such as Physics-Informed Neural Networks (PINN), multi-task learning, etc.), there is a lack of effective metrics to objectively compare the performance of these methods.
- The authors propose using Kolmogorov n-width as a metric to evaluate different PIML architectures, and use it to analyze, benchmark, and improve models.

2. **Addressing Uncertainty in Training**:
- Issues such as overfitting and selective sampling during training lead to increased uncertainty during model validation.
- By introducing the Kolmogorov n-width metric, this uncertainty can be reduced and the model's generalization ability can be improved.

3. **Optimizing the Choice of Activation Functions**:
- Different activation functions have a significant impact on the model's generalization ability, especially when dealing with "worst-case" scenarios.
- The authors found through experiments that different activation functions (such as sine functions and hyperbolic tangent functions) have different impacts on the model's generalization ability and proposed improvements.

4. **Improving Model Generalization Ability**:
- By incorporating the Kolmogorov n-width metric into the optimization process, the model's generalization ability on multi-task PDE problems can be improved.

In summary, this paper aims to provide a new metric—Kolmogorov n-width—for comparing and improving existing PIML methods, especially in multi-task learning scenarios, to enhance model generalization ability and reduce uncertainty during the training process.