Abstract:We present an efficient and scalable algorithm for performing matrix-vector multiplications ("matvecs") for block Toeplitz matrices. Such matrices, which are shift-invariant with respect to their blocks, arise in the context of solving inverse problems governed by autonomous systems, and time-invariant systems in particular. In this article, we consider inverse problems that are solved for inferring unknown parameters from observational data of a linear time-invariant dynamical system given in the form of partial differential equations (PDEs). Matrix-free Newton-conjugate-gradient methods are often the gold standard for solving these inverse problems, but they require numerous actions of the Hessian on a vector. Matrix-free adjoint-based Hessian matvecs require solution of a pair of linearized forward/adjoint PDE solves per Hessian action, which may be prohibitive for large-scale inverse problems, especially when efficient low-rank approximations of the Hessian are not readily available, such as for hyperbolic PDE operators. Time invariance of the forward PDE problem leads to a block Toeplitz structure of the discretized parameter-to-observable (p2o) map defining the mapping from inputs (parameters) to outputs (observables) of the PDEs. This block Toeplitz structure enables us to exploit two key properties: (1) compact storage of the p2o map and its adjoint; and (2) efficient fast Fourier transform (FFT)-based Hessian matvecs. The proposed algorithm is mapped onto large multi-GPU clusters and achieves more than 80 percent of peak bandwidth on an NVIDIA A100 GPU. Excellent weak scaling is shown for up to 48 A100 GPUs. For the targeted problems, the implementation executes Hessian matvecs within fractions of a second, orders of magnitude faster than can be achieved by the conventional matrix-free Hessian matvecs via forward/adjoint PDE solves.

Computing matrix trigonometric functions with GPUs through Matlab

A GEMM interface and implementation on NVIDIA GPUs for multiple small matrices

Fast and Scalable FFT-Based GPU-Accelerated Algorithms for Hessian Actions Arising in Linear Inverse Problems Governed by Autonomous Dynamical Systems

GPU Accelerated Newton for Taylor Series Solutions of Polynomial Homotopies in Multiple Double Precision

High Performance Matrix Multiplication on General Purpose Graphics Processing Units

Acceleration of Approximate Matrix Multiplications on GPUs

A fast, dense Chebyshev solver for electronic structure on GPUs

Fast hardware-aware matrix-free algorithms for higher-order finite-element discretized matrix multivector products on distributed systems

Fast hardware-aware matrix-free algorithm for higher-order finite-element discretized matrix multivector products on distributed systems

Batched Kronecker product for 2-D matrices and 3-D arrays on NVIDIA GPUs

Tridiagonalization of a dense symmetric matrix on multiple GPUs and its application to symmetric eigenvalue problems

Treecode and fast multipole method for N-body simulation with CUDA

Accelerating Algebraic Multigrid Solvers on NVIDIA GPUs

Accelerating Iterative Linear Solvers Using Multiple Graphical Processing Units

Accelerating Pythonic coupled cluster implementations: a comparison between CPUs and GPUs

A simple GPU implementation of spectral-element methods for solving 3D Poisson type equations on rectangular domains and its applications

Design and Implementation of Matrix Multiplication on GPU

An adaptive finite element multigrid solver using GPU acceleration

Performant low-order matrix-free finite element kernels on GPU architectures

AmgX: A Library for GPU Accelerated Algebraic Multigrid and Preconditioned Iterative Methods

KSSOLV-GPU: an Efficient GPU-enabled MATLAB Toolbox for Solving the Kohn-Sham Equations Within Density Functional Theory in Plane-Wave Basis Set