Abstract:Foundation models have revolutionized language modeling, while whether this success is replicated in scientific computing remains unexplored. We present OmniArch, the first prototype aiming at solving multi-scale and multi-physics scientific computing problems with physical alignment. We addressed all three challenges with one unified architecture. Its pre-training stage contains a Fourier Encoder-decoder fading out the disharmony across separated dimensions and a Transformer backbone integrating quantities through temporal dynamics, and the novel PDE-Aligner performs physics-informed fine-tuning under flexible conditions. As far as we know, we first conduct 1D-2D-3D united pre-training on the PDEBench, and it sets not only new performance benchmarks for 1D, 2D, and 3D PDEs but also demonstrates exceptional adaptability to new physics via in-context and zero-shot learning approaches, which supports realistic engineering applications and foresight physics discovery.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to construct a fundamental model capable of handling multi - scale and multi - physics problems in scientific computing. Specifically, the paper proposes the OmniArch model, aiming to address the following three main challenges through physical alignment: 1. **Multi - scale problems**: Handle inputs of different dimensions (1D, 2D, 3D), different grid resolutions, and different grid shapes. For example, fluid - dynamics simulations can range from simple 1D pipe flows to complex 3D turbulent flows, and the model needs to maintain accuracy and consistency across these different scales. 2. **Multi - physics problems**: Handle dynamic systems involving different physical quantities. For example, in meteorology, multiple physical quantities such as wind speed, temperature, and humidity interact, and the model needs to handle these different physical fields simultaneously. 3. **Physical alignment problems**: Flexibly incorporate physical prior knowledge (such as governing equations, symmetries, conservation laws, and boundary conditions) into the solution process. For example, in heat conduction problems, the law of energy conservation and boundary conditions are crucial for predicting temperature distributions. To meet these challenges, OmniArch adopts the following methods: - **Pre - training stage**: - Use the Fourier Encoder - decoder to handle data of different dimensions and form PDE token embeddings by truncating frequency patterns. These embeddings are arranged in time steps and processed by the Transformer module. - Introduce the Temporal Mask mechanism to ensure that each physical quantity can simultaneously focus on all physical quantities and previous time steps, thereby achieving effective learning of multi - physics systems. - **Fine - tuning stage**: - Develop the PDE - Aligner for fine - tuning of physical alignment, utilize the hidden representations of equations and other physical priors to align the prediction results with the known physical field dynamics and improve the consistency of the model with traditional physical constraints. Through these methods, OmniArch not only achieves state - of - the - art performance in multiple PDE benchmark tests but also demonstrates strong generalization ability in zero - shot learning and in - context learning tasks. In addition, OmniArch supports multi - scale inference and can make predictions on input grids of different resolutions while maintaining high accuracy.

Building Flexible Machine Learning Models for Scientific Computing at Scale

Scientific Machine Learning Enables Multiphysics Digital Twins of Large-Scale Electronic Chips

Universal and High-Fidelity Resolution Extending for Fluorescence Microscopy Using a Single-Training Physics-Informed Sparse Neural Network

OmniJet-$α$: The first cross-task foundation model for particle physics

OmniBal: Towards Fast Instruct-tuning for Vision-Language Models via Omniverse Computation Balance

Differentiable Multi-Fidelity Fusion: Efficient Learning of Physics Simulations with Neural Architecture Search and Transfer Learning

3D-Transformer: Molecular Representation with Transformer in 3D Space

Earthfarsser: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model

Fast and flexible range-separated models for atomistic machine learning

The confluence of machine learning and multiscale simulations

Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model

DPA-2: a large atomic model as a multi-task learner

Optical training of large-scale Transformers and deep neural networks with direct feedback alignment

Integrating Machine Learning with Physics-Based Modeling

OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System

Pretraining Billion-scale Geospatial Foundational Models on Frontier

MaD-Scientist: AI-based Scientist solving Convection-Diffusion-Reaction Equations Using Massive PINN-Based Prior Data

Interfacing finite elements with deep neural operators for fast multiscale modeling of mechanics problems

Differentiable modeling to unify machine learning and physical models and advance Geosciences

Comprehensive Performance Modeling and System Design Insights for Foundation Models