Abstract:Single-operator learning involves training a deep neural network to learn a specific operator, whereas recent work in multi-operator learning uses an operator embedding structure to train a single neural network on data from multiple operators. Thus, multi-operator learning is capable of predicting a range of operators within one model. In this work, we propose pretraining and fine-tuning strategies for solving PDEs using multi-operator learning. One key aspect is that by increasing the number of families of operators used in pretraining, a PDE foundation model can be fine-tuned to downstream tasks involving new PDEs with a limited number of samples, thus outperforming single operator neural networks. Specifically, a multi-operator learning model pre-trained with data from diverse PDE families can predict unseen operators after fine-tuning with only a limited number of operators from the new family, enabling them to serve as a data-free PDE solver. We also show that the proposed training and fine-tuning method is able to predict new operators in zero-shot prediction without samples. Additionally, we introduce a PDE-agnostic meta-learning algorithm to improve the adaptability of the model to various PDEs by providing a better parameter initialization process. To address the needs of applications with limited computing resources, we explore low-rank adaptation methods that reduce computational costs while enhancing solver accuracy. Lastly, by examining the scaling law with respect to the number of operator families, we establish and highlight its potential for broad adaptation in PDE-solving tasks.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address several key issues in solving partial differential equations (PDEs), particularly how to leverage Multi-Operator Learning (MOL) to improve model performance when handling new tasks. #### Core Issues: 1. **Generalization Ability**: Traditional Single-Operator Learning (SOL) methods perform poorly when dealing with new equations or new input function distributions. This paper proposes a Multi-Operator Learning approach that uses pre-training and fine-tuning strategies to better adapt the model to new tasks. 2. **Adaptability with Limited Data**: How to maintain high accuracy when only a small amount of data is available for new tasks. The paper demonstrates that by pre-training a model with various operator families and fine-tuning it with a small amount of new data, this goal can be achieved. 3. **Zero-Shot Prediction**: Whether the model can accurately predict new operators it has not seen before without additional samples. The paper proposes a framework that allows the model to achieve zero-shot prediction with minimal sample fine-tuning after pre-training. #### Methodology: - **Pre-training and Fine-tuning Strategy**: Using a large number of operator families for pre-training and then fine-tuning for specific new tasks to improve the model's generalization ability. - **Low-Rank Adaptation Technique**: Employing Low-Rank Adaptation (LoRA) to enhance the efficiency and accuracy of model fine-tuning. - **Meta-Learning Algorithm**: Introducing a PDE-agnostic meta-learning algorithm to improve the model's rapid adaptability to new tasks. #### Main Contributions: 1. **Fine-tuning Strategy**: Proposing a method to fine-tune MOL models using embedded structures. Experiments show that increasing the number of different operator families used in pre-training can significantly improve the model's accuracy with limited data. 2. **Zero-Shot Prediction**: Demonstrating that pre-trained models can achieve zero-shot prediction for unseen new operators with only minimal sample fine-tuning. 3. **Meta-Learning and Low-Rank Fine-tuning**: Exploring meta-learning pre-training strategies and low-rank fine-tuning techniques for MOL models to further optimize the entire pre-training-fine-tuning process. In summary, the main objective of this paper is to propose a new Multi-Operator Learning framework, LeMON-PROSE, to address the generalization ability and adaptability issues of traditional Single-Operator Learning methods when facing new tasks with limited data, and to showcase its potential in zero-shot prediction.

LeMON: Learning to Learn Multi-Operator Networks

DeepONet as a Multi-Operator Extrapolation Model: Distributed Pretraining with Physics-Informed Fine-Tuning

MODNO: Multi Operator Learning With Distributed Neural Operators

Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning

Latent Neural Operator Pretraining for Solving Time-Dependent PDEs

Synergistic Learning with Multi-Task DeepONet for Efficient PDE Problem Solving

MIONet: Learning multiple-input operators via tensor product

Data Generation-based Operator Learning for Solving Partial Differential Equations on Unbounded Domains

Learning Partial Differential Equations with Deep Parallel Neural Operator

DeepONet for Solving PDEs: Generalization Analysis in Sobolev Training

Separable Operator Networks

One-shot learning for solution operators of partial differential equations

M$^{2}$M: Learning controllable Multi of experts and multi-scale operators are the Partial Differential Equations need

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators

Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolation

Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems

M^2M: Learning Controllable Multi of Experts and Multi-Scale Operators Are the Partial Differential Equations Need

MetaNO: How to Transfer Your Knowledge on Learning Hidden Physics

In-context operator learning with data prompts for differential equation problems

PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamics