Abstract:Recent research has shown that collaborative computing of CPUs and GPUs in the same system can effectively accelerate large-scale SGD-based matrix factorization (MF), but it faces the problem of limited scalability due to parameter synchronization in the server. Theoretically, asynchronous methods can overcome this shortcoming. However, through a series of tests, observations, and analyses, we realize that developing an effective asynchronous multi-CPU/GPU MF framework faces several major design challenges: the underutilized CPUs, high communication overhead, and the asynchronous data safety issue. This article presents a unified multi-CPU/GPU asynchronous computing framework for SGD-based matrix factorization, named UMA-MF. UMA-MF treats CPUs and GPUs in the system as distributed workers that train matrix datasets in parallel and update feature parameters asynchronously. It provides a cache-friendly CPU external working mode, which can improve the CPU's cache hit rate, thereby promoting the efficient use of CPUs. It offers an algorithm to find the shortest communication ring topology of heterogeneous CPU/GPU workers and builds computing-communication pipelines to minimize the communication overhead. It implements a wait-free structure and load-balanced data distribution to achieve asynchronous data safety. UMA-MF can effectively accelerate SGD-based MF on multi-CPU/GPU systems in an asynchronous way. On a physical platform with configurations ranging from single processor system to 2CPUs--4CPUs system, for five common datasets Netfix, R1, R2, Goodreads, and de-dense, UMA-MF achieves up to 3.56x speedup compared with HCC-MF, which is the state-of-the-art multi-CPU/GPU synchronous computing framework for SGD-based MF. UMA-MF also shows good scalability. When the system is scaled to 2CPUs-4GPUs, the training time speedup of UMA-MF can reach 70%--97% of the ideal speedup.

AEML: An Acceleration Engine for Multi-GPU Load-balancing in Distributed Heterogeneous Environment

Automatic execution of single-GPU computations across multiple GPUs

A Virtual Multi-Channel GPU Fair Scheduling Method for Virtual Machines.

Multi-GPU Hybrid Programming Accelerated Three-Dimensional Phase-Field Model in Binary Alloy

Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms

Dynamic Load Balancing in GPU-Based Systems - Early Experiments

An optimized large-scale hybrid DGEMM design for CPUs and ATI GPUs.

HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models

A Self-Adaptive Workload Balancing Algorithm on Gpu Clusters

Towards Large-Scale Evolutionary Multi-Tasking: A GPU-Based Paradigm

Fela: Incorporating Flexible Parallelism and Elastic Tuning to Accelerate Large-Scale DML

Deep Reinforcement Learning Based Computation Offloading in Heterogeneous MEC Assisted by Ground Vehicles and Unmanned Aerial Vehicles

Efficient GPU Spatial-Temporal Multitasking

Orchestrated Co-scheduling, Resource Partitioning, and Power Capping on CPU-GPU Heterogeneous Systems via Machine Learning

GPU-HADVPPM4HIP V1.0: using the heterogeneous-compute interface for portability (HIP) to speed up the piecewise parabolic method in the CAMx (v6.10) air quality model on China's domestic GPU-like accelerator

Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers

Design and realization of hybrid resource management system for heterogeneous cluster

UMA-MF: A Unified Multi-CPU/GPU Asynchronous Computing Framework for SGD-Based Matrix Factorization

3M-AI: A Multi-task and Multi-core Virtualization Framework for Multi-FPGA AI Systems in the Cloud

AMOEBA: A Coarse Grained Reconfigurable Architecture for Dynamic GPU Scaling

Load Balancing of Multi-Core Heterogeneous CPU Based on BP Neural Network and Weighted Round-Robin Algorithm