Abstract:Scientific computing is an essential tool for scientific discovery and engineering design, and its computational cost is always a main concern in practice. To accelerate scientific computing, it is a promising approach to use machine learning (especially meta-learning) techniques for selecting hyperparameters of traditional numerical methods. There have been numerous proposals to this direction, but many of them require automatic-differentiable numerical methods. However, in reality, many practical applications still depend on well-established but non-automatic-differentiable legacy codes, which prevents practitioners from applying the state-of-the-art research to their own problems. To resolve this problem, we propose a non-intrusive methodology with a novel gradient estimation technique to combine machine learning and legacy numerical codes without any modification. We theoretically and numerically show the advantage of the proposed method over other baselines and present applications of accelerating established non-automatic-differentiable numerical solvers implemented in PETSc, a widely used open-source numerical software library.

What problem does this paper attempt to address?

This paper focuses on accelerating the execution of legacy numerical solvers, particularly those that do not support automatic differentiation. Currently, machine learning (especially meta-learning) techniques are used to optimize hyperparameters of numerical methods to improve the speed of scientific computing. However, many real-world applications rely on non-automatic differentiation legacy code, limiting the application of the latest research. The paper proposes a non-intrusive gradient-based meta-solving method (NI-GBMS) that uses an adaptive surrogate model to estimate gradients, allowing the integration of machine learning with legacy numerical solvers without modifying the original code. This method is theoretically and numerically superior to other baselines and demonstrates acceleration effects on non-automatic differentiation numerical solvers in widely used open-source numerical software libraries like PETSc. Specific contributions of the paper include: 1. Introducing a non-intrusive method to train neural networks (meta-solvers) that accelerate legacy numerical solvers running in conjunction with non-automatic differentiation black box programs. 2. Improving the convergence speed of training by reducing the variance of forward gradients through the use of adaptive surrogate models for controlling variations. 3. Successfully applying the proposed method to non-automatic differentiation numerical solvers in PETSc, achieving significant acceleration. This is the first successful attempt to train neural networks as meta-solvers in conjunction with legacy solvers. The paper points out that a major limitation of the current joint training methods is the requirement of gradients of numerical solver outputs with respect to their parameters, which is often impractical in many real-world applications. To address this issue, they use a novel gradient estimation technique that is compatible with existing code and does not require any modifications.

Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solving

Principled Acceleration of Iterative Numerical Methods Using Machine Learning

A machine learning framework for data driven acceleration of computations of differential equations

AI‐enhanced iterative solvers for accelerating the solution of large‐scale parametrized systems

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Efficient and stable SAV-based methods for gradient flows arising from deep learning

A fast neural hybrid Newton solver adapted to implicit methods for nonlinear dynamics

Automatic discovery of optimal meta-solvers via multi-objective optimization

Differentiable Programming for Computational Plasma Physics

The Stabilized Explicit Variable-Load Solver with Machine Learning Acceleration for the Rapid Solution of Stiff Chemical Kinetics

Parallelisable computation of the gradient in nonlinear stochastic optimal control problems

Learning to optimize with convergence guarantees using nonlinear system theory

Meta-PDE: Learning to Solve PDEs Quickly Without a Mesh

Symplectic Extra-gradient Type Method for Solving General Non-monotone Inclusion Problem

Learning to Accelerate Partial Differential Equations via Latent Global Evolution

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes

Accelerated Sparse Recovery Via Gradient Descent with Nonlinear Conjugate Gradient Momentum

Speeding up and reducing memory usage for scientific machine learning via mixed precision

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Solving inverse problems in physics by optimizing a discrete loss: Fast and accurate learning without neural networks