Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators

Benedikt Alkin,Andreas Fürst,Simon Schmid,Lukas Gruber,Markus Holzleitner,Johannes Brandstetter

2024-10-10

Abstract:Neural operators, serving as physics surrogate models, have recently gained increased interest. With ever increasing problem complexity, the natural question arises: what is an efficient way to scale neural operators to larger and more complex simulations - most importantly by taking into account different types of simulation datasets. This is of special interest since, akin to their numerical counterparts, different techniques are used across applications, even if the underlying dynamics of the systems are similar. Whereas the flexibility of transformers has enabled unified architectures across domains, neural operators mostly follow a problem specific design, where GNNs are commonly used for Lagrangian simulations and grid-based models predominate Eulerian simulations. We introduce Universal Physics Transformers (UPTs), an efficient and unified learning paradigm for a wide range of spatio-temporal problems. UPTs operate without grid- or particle-based latent structures, enabling flexibility and scalability across meshes and particles. UPTs efficiently propagate dynamics in the latent space, emphasized by inverse encoding and decoding techniques. Finally, UPTs allow for queries of the latent space representation at any point in space-time. We demonstrate diverse applicability and efficacy of UPTs in mesh-based fluid simulations, and steady-state Reynolds averaged Navier-Stokes simulations, and Lagrangian-based dynamics.

Machine Learning,Artificial Intelligence,Fluid Dynamics

What problem does this paper attempt to address?

The paper aims to address the scalability issues of neural operators when dealing with large-scale and complex simulations. Specifically, the paper proposes Universal Physics Transformers (UPTs), an efficient and unified learning paradigm suitable for a wide range of spatiotemporal problems. The main objectives of UPTs include: 1. **Unified Architecture**: By introducing a latent structure that is independent of grids or particles, UPTs achieve flexibility and scalability across different simulation datasets. 2. **Efficient Propagation of Dynamic Information**: Utilizing inverse encoding and decoding techniques, UPTs efficiently propagate dynamic information in the latent space and allow querying of latent representations at any spatiotemporal point. 3. **Adaptation to Different Types of Simulation Datasets**: UPTs can flexibly encode different grids or varying numbers of particles into a unified latent representation, thereby enabling scalability to large-scale systems. In summary, the main purpose of this paper is to enhance the efficiency and generality of neural operators in handling large-scale and complex physical simulations by introducing UPTs.

Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators

Multiscale Neural Operator: Learning Fast and Grid-independent PDE Solvers

Physics-enhanced Neural Operator for Simulating Turbulent Transport

Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEs

Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity

Neural Operators for Accelerating Scientific Simulations and Design

Multi-scale Time-stepping of Partial Differential Equations with Transformers

HT-Net: Hierarchical Transformer Based Operator Learning Model for Multiscale PDEs

UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation

Learning Physical Simulation with Message Passing Transformer

Physics-Informed Deep Neural Operator Networks

PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks

A physics-informed transformer neural operator for learning generalized solutions of initial boundary value problems

Physics Informed Token Transformer for Solving Partial Differential Equations

Learning nonlinear operators in latent spaces for real-time predictions of complex dynamics in physical systems

GNOT: A General Neural Operator Transformer for Operator Learning

Lorentz-Equivariant Geometric Algebra Transformers for High-Energy Physics

Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks and Operators in Scientific Computing: Fluid and Solid Mechanics

Metamizer: a versatile neural optimizer for fast and accurate physics simulations

Transolver: A Fast Transformer Solver for PDEs on General Geometries

Exploring Efficient Partial Differential Equation Solution Using Speed Galerkin Transformer