Abstract:Multi-fidelity surrogate modeling aims to learn an accurate surrogate at the highest fidelity level by combining data from multiple sources. Traditional methods relying on Gaussian processes can hardly scale to high-dimensional data. Deep learning approaches utilize neural network based encoders and decoders to improve scalability. These approaches share encoded representations across fidelities without including corresponding decoder parameters. This hinders inference performance, especially in out-of-distribution scenarios when the highest fidelity data has limited domain coverage. To address these limitations, we propose Multi-fidelity Residual Neural Processes (MFRNP), a novel multi-fidelity surrogate modeling framework. MFRNP explicitly models the residual between the aggregated output from lower fidelities and ground truth at the highest fidelity. The aggregation introduces decoders into the information sharing step and optimizes lower fidelity decoders to accurately capture both in-fidelity and cross-fidelity information. We show that MFRNP significantly outperforms state-of-the-art in learning partial differential equations and a real-world climate modeling task. Our code is published at: <a class="link-external link-https" href="https://github.com/Rose-STL-Lab/MFRNP" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address several key issues in multi-fidelity surrogate modeling: 1. **Scalability with High-Dimensional Data**: Traditional Gaussian Processes (GPs)-based methods struggle to scale with high-dimensional data. While deep learning methods improve scalability by using neural network encoders and decoders, these methods typically only use the representations generated by the encoder for cross-fidelity information sharing, neglecting the corresponding decoder parameters. 2. **Accuracy of Cross-Fidelity Information Sharing**: Existing methods mainly rely on the latent representations generated by the encoder for cross-fidelity information sharing, without incorporating the decoder parameters into the sharing process. This limits the model's inference performance, especially when dealing with inputs that are out-of-distribution (OOD) from the training data, particularly in scenarios with limited coverage of the highest fidelity data domain. 3. **Optimization of Residual Modeling**: To improve prediction accuracy at the highest fidelity, it is essential to effectively model the residuals between low-fidelity outputs and the true values. Current methods fall short in this aspect. To address the above issues, the authors propose the Multi-fidelity Residual Neural Processes (MFRNP), a novel multi-fidelity surrogate modeling framework. MFRNP explicitly models the residuals between the aggregated low-fidelity outputs and the highest fidelity true values, and introduces the decoder in the information sharing step, thereby enhancing the model's accuracy and scalability. Additionally, the authors design a new evidence lower bound (Residual-ELBO) to optimize cross-fidelity information sharing and residual modeling. ### Main Contributions 1. **Novel Multi-fidelity Surrogate Model**: The MFRNP architecture can share input-specific information from low fidelity, addressing the error propagation issues caused by different decoder parameters while maintaining scalability. 2. **New Loss Function**: The Residual-ELBO promotes cross-fidelity learning and optimizes the low-fidelity decoder for residual modeling at the highest fidelity. 3. **Outstanding Performance**: MFRNP demonstrates superior performance in large-scale global climate modeling and multiple partial differential equation (PDE) benchmark tasks, with an average performance improvement of approximately 90% over existing state-of-the-art baseline models. ### Experimental Validation The authors conducted experiments on multiple benchmark tasks, including partial differential equations (PDEs), fluid simulation, and earth surface temperature prediction. The experimental results show that MFRNP outperforms other methods across various settings, particularly excelling in handling inputs that are out-of-distribution from the training data.

Multi-Fidelity Residual Neural Processes for Scalable Surrogate Modeling

Multi-fidelity Hierarchical Neural Processes

Deep Residual Surrogate Model

Residual Multi-Fidelity Neural Network Computing

Residual Gaussian Process: A Tractable Nonparametric Bayesian Emulator for Multi-Fidelity Simulations

A Multi-Fidelity Surrogate Modeling Approach for Incorporating Multiple Non-Hierarchical Low-Fidelity Data

Multi-fidelity reduced-order surrogate modelling

A Single-Fidelity Surrogate Modeling Method Based on Nonlinearity Integrated Multi-Fidelity Surrogate

Multi-fidelity Gaussian process surrogate modeling for regression problems in physics

A novel multi-fidelity surrogate modeling framework integrated with sequential sampling criterion for non-hierarchical data

Disentangled Multi-Fidelity Deep Bayesian Active Learning

Surrogate modelling and uncertainty quantification based on multi-fidelity deep neural network

A Multi-Output Multi-Fidelity Gaussian Process Model for Non-Hierarchical Low-Fidelity Data Fusion

Multi-fidelity Surrogate Modeling for Application/Architecture Co-design

A new multi-fidelity surrogate modelling method for engineering design based on neural network and transfer learning

Multi-fidelity Fourier Neural Operator for Fast Modeling of Large-Scale Geological Carbon Storage

A comprehensive multi-fidelity surrogate framework based on Gaussian process for datasets with heterogeneous responses

A multi-fidelity surrogate modeling method in the presence of non-hierarchical low-fidelity data

General multi-fidelity surrogate models: Framework and active learning strategies for efficient rare event simulation

A Bayesian neural network approach to Multi-fidelity surrogate modelling

Active Learning-Assisted Multi-Fidelity Surrogate Modeling Based on Geometric Transformation