Abstract:In the field of computational science and engineering, solving nonlinear transient problems still poses a challenging task that often requires significant computational resources. This research introduces a novel methodology that harnesses the power of cutting-edge Temporal Fusion Transformers (TFTs) to accelerate the solution of such problems in multi-query scenarios (i.e. parameterized problems). At each time step, TFT models, renowned for their time series forecasting capabilities, are combined with dimensionality reduction techniques to efficiently generate initial solutions for nonlinear solvers. Specifically, during the training phase, a reduced set of high-fidelity system solutions is obtained by solving the system of differential equations governing the problem for different parameter instances. Then, dimensionality reduction is applied to create a reduced latent space to simplify the representation of the complex system solutions. Subsequently, TFT models are trained for one-step-ahead forecasting in the latent space, utilizing information from previous states to make accurate predictions about future states. The TFTs' predictions are fed back to the system as initial guesses at each time step of the solution algorithm and are then guided towards the exact solutions that satisfy equilibrium using Newton–Raphson (NR) iterations. The basic premise of the proposed idea is that having accurate initial predictions will significantly decrease the number of the costly NR-iterations needed in nonlinear dynamic problems, effectively reducing the solution time. In addition, the proposed scheme is able to handle problems with parametric and time-variant forcing forces. A customized TFT architecture is developed that takes as input not only the response history of the system, but also the current loading, and makes informed guesses about the future state of the system. The methodology's effectiveness is demonstrated in numerical applications that involve high nonlinearity, where the TFT-generated initial solutions resulted in a notable reduction in the number of NR-iterations required for solver convergence. This significant enhancement in computational efficiency holds substantial promise, especially in scenarios involving a multitude of analyses and high iteration demands, with wide-ranging applications across computational mechanics and related fields.

Multi-scale Time-stepping of Partial Differential Equations with Transformers

Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity

Exploring Efficient Partial Differential Equation Solution Using Speed Galerkin Transformer

Scalable Transformer for PDE Surrogate Modeling

Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEs

Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equations

Self-supervised Pretraining for Partial Differential Equations

Predicting Ordinary Differential Equations with Transformers

Non-intrusive Surrogate Modeling for Parametrized Time-dependent PDEs using Convolutional Autoencoders

Physics Informed Token Transformer for Solving Partial Differential Equations

A transformer-based neural ODE for dense prediction

HT-Net: Hierarchical Transformer Based Operator Learning Model for Multiscale PDEs

Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators

HAMLET: Graph Transformer Neural Operator for Partial Differential Equations

Physics‐informed surrogates for electromagnetic dynamics using Transformers and graph neural networks

Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting

Learning stochastic dynamics and predicting emergent behavior using transformers

Learning Partial Differential Equations with Deep Parallel Neural Operator

Neural-PDE: a RNN Based Neural Network for Solving Time Dependent PDEs

Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting

Fusing nonlinear solvers with transformers for accelerating the solution of parametric transient problems