Abstract:The generalized Dryja--Smith--Widlund (GDSW) preconditioner is a two-level overlapping Schwarz domain decomposition (DD) preconditioner that couples a classical one-level overlapping Schwarz preconditioner with an energy-minimizing coarse space. When used to accelerate the convergence rate of Krylov subspace iterative methods, the GDSW preconditioner provides robustness and scalability for the solution of sparse linear systems arising from the discretization of a wide range of partial different equations. In this paper, we present FROSch (Fast and Robust Schwarz), a domain decomposition solver package which implements GDSW-type preconditioners for both CPU and GPU clusters. To improve the solver performance on GPUs, we use a novel decomposition to run multiple MPI processes on each GPU, reducing both solver's computational and storage costs and potentially improving the convergence rate. This allowed us to obtain competitive or faster performance using GPUs compared to using CPUs alone. We demonstrate the performance of FROSch on the Summit supercomputer with NVIDIA V100 GPUs, where we used NVIDIA Multi-Process Service (MPS) to implement our decomposition strategy. The solver has a wide variety of algorithmic and implementation choices, which poses both opportunities and challenges for its GPU implementation. We conduct a thorough experimental study with different solver options including the exact or inexact solution of the local overlapping subdomain problems on a GPU. We also discuss the effect of using the iterative variant of the incomplete LU factorization and sparse-triangular solve as the approximate local solver, and using lower precision for computing the whole FROSch preconditioner. Overall, the solve time was reduced by factors of about $2\times$ using GPUs, while the GPU acceleration of the numerical setup time depend on the solver options and the local matrix sizes.

An Efficient Approach of GPU-accelerated Stochastic Gradient Descent Method for Matrix Factorization

CuMF_SGD: Parallelized Stochastic Gradient Descent for Matrix Factorization on GPUs.

GPU accelerated matrix factorization of large scale data using block based approach

A Fast Parallel Stochastic Gradient Method for Matrix Factorization in Shared Memory Systems

CuMF_SGD: Fast and Scalable Matrix Factorization.

UMA-MF: A Unified Multi-CPU/GPU Asynchronous Computing Framework for SGD-Based Matrix Factorization

Alternating Mixing Stochastic Gradient Descent for Large-scale Matrix Factorization

Efficient GPU implementation of randomized SVD and its applications

Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Optimization of Sparse Matrix Computation for Algebraic Multigrid on GPUs

Accelerating Matrix Factorization by Dynamic Pruning for Fast Recommendation

An Experimental Study of Two-Level Schwarz Domain Decomposition Preconditioners on GPUs

A Novel Multi-CPU/GPU Collaborative Computing Framework for SGD-based Matrix Factorization

GPU Accelerated Sparse Cholesky Factorization

Large-scale and Scalable Latent Factor Analysis via Distributed Alternative Stochastic Gradient Descent for Recommender Systems

Large-Scale Distributed Bayesian Matrix Factorization using Stochastic Gradient MCMC

Optimizing sparse matrix-vector multiplication based on gpu

Stochastic Gradient Descent for matrix completion: Hybrid parallelization on shared- and distributed-memory systems

Fast Asynchronous Parallel Stochastic Gradient Decent

Hardware-Efficient Mixed-Precision CP Tensor Decomposition

A New Hybrid GPU-CPU Sparse LDLT Factorization Algorithm with GPU and CPU Factorizing Concurrently