Abstract:The sparse matrix-vector multiplication (SpMV) is of great importance in scientific computations. Graphics processing unit (GPU)-accelerated SpMVs for large-sized problems have attracted considerable attention recently. We observe that on a specific multi-GPU platform, the SpMV performance can usually be greatly improved when a matrix is partitioned into several blocks according to a predetermined rule and each block is assigned to a GPU with an appropriate storage format. This motivates us to propose a novel multi-GPU parallel SpMV optimization framework, which involves the following parts: (1) a simple rule is defined to divide any given matrix among multiple GPUs; (2) a performance model, which is independent of the problems and dependent on the resources of devices, is proposed to accurately predict the execution time of SpMV kernels; and (3) a selection algorithm is suggested to automatically select the most appropriate one from the storage formats that are involved in the framework for the matrix block that is assigned to each GPU on the basis of the performance model. The objective of our framework does not construct a new storage format or algorithm but automatically and rapidly generates an optimally parallel SpMV for any sparse matrix on a specific multi-GPU platform by integrating the existing storage formats and their corresponding kernels. We take 5 popular storage formats, for example, to present the idea of constructing the framework. Theoretically, we validate the correctness of our proposed SpMV performance model. This model is constructed only once for each type of GPU. Moreover, this framework is general and easy to be extensible. For a storage format that is not included in our framework, once the performance model of its corresponding SpMV kernel is successfully constructed, it can be incorporated into our framework. The experiments validate the efficiency of our proposed framework.

UMA-MF: A Unified Multi-CPU/GPU Asynchronous Computing Framework for SGD-Based Matrix Factorization

A Novel Multi-CPU/GPU Collaborative Computing Framework for SGD-based Matrix Factorization

CuMF_SGD: Fast and Scalable Matrix Factorization.

Efficient Matrix Factorization on Heterogeneous CPU-GPU Systems

CuMF_SGD: Parallelized Stochastic Gradient Descent for Matrix Factorization on GPUs.

MSGD: A Novel Matrix Factorization Approach for Large-Scale Collaborative Filtering Recommender Systems on GPUs.

GPUMF: A GPU-Enpowered Collaborative Filtering Algorithm Through Matrix Factorization

CuSNMF: A Sparse Non-Negative Matrix Factorization Approach for Large-Scale Collaborative Filtering Recommender Systems on Multi-GPU

An Efficient Parallelization Approach for Large-Scale Sparse Non-Negative Matrix Factorization Using Kullback-Leibler Divergence on Multi-GPU.

GPU accelerated matrix factorization of large scale data using block based approach

An Efficient Approach of GPU-accelerated Stochastic Gradient Descent Method for Matrix Factorization

ApSpGEMM: Accelerating Large-scale SpGEMM with Heterogeneous Collaboration and Adaptive Panel

An Efficient Parallel Stochastic Gradient Descent for Matrix Factorization on GPUS

An Online and Scalable Model for Generalized Sparse Nonnegative Matrix Factorization in Industrial Applications on Multi-GPU

Clmf: A Fine-Grained and Portable Alternating Least Squares Algorithm for Parallel Matrix Factorization

Parallel multiple nonnegative matrices factorization using graphics processing unit

A novel multi-graphics processing unit parallel optimization framework for the sparse matrix-vector multiplication.

Rima: An RDMA-Accelerated Model-Parallelized Solution to Large-Scale Matrix Factorization

An Asynchronous Parallel Implementation of Multilevel Fast Multipole Algorithm on GPU Cluster for 3D Electromagnetic Scattering Problems

High Performance Coordinate Descent Matrix Factorization for Recommender Systems.

Distributed Out-of-Memory NMF on CPU/GPU Architectures