Abstract:Modern computing platforms tend to deploy multiple GPUs (2, 4, or more) on a single node to boost system performance, with each GPU having a large capacity of global memory and streaming multiprocessors (SMs). GPUs are an expensive resource, and boosting utilization of GPUs without causing performance degradation of individual workloads is an important and challenging problem. Although services like MPS support simultaneous execution of multiple co-operative kernels on a single device, they do not solve the above problem for uncooperative kernels, MPS being oblivious to the resource needs of each kernel. We propose a fully automated compiler-assisted scheduling framework. The compiler constructs GPU tasks by identifying kernel launches and their related GPU operations (e.g. memory allocations). For each GPU task, a probe is instrumented in the host-side code right before its launch point. At runtime, the probe conveys the information about the task's resource requirements (e.g. memory and compute cores) to a scheduler, such that the scheduler can place the task on an appropriate device based on the task's resource requirements and devices' load in a memory-safe, resource-aware manner. To demonstrate its advantages, we prototyped a throughput-oriented scheduler based on the framework, and evaluated it with the Rodinia benchmark suite and the Darknet neural network framework on NVIDIA GPUs. The results show that the proposed solution outperforms existing state-of-the-art solutions by leveraging its knowledge about applications' multiple resource requirements, which include memory as well as SMs. It improves throughput by up to 2.5x for Rodinia benchmarks, and up to 2.7x for Darknet neural networks. In addition, it improves job turnaround time by up to 4.9x, and limits individual kernel performance degradation to at most 2.5%.

Can Flexible Multi-Core Scheduling Help to Execute Machine Learning Algorithms Resource-Efficiently?

An Optimized RM Algorithm by Task Affinity on Multi-Core Processor

Efficient Tasks Scheduling in Multicore Systems Integrated with Hardware Accelerators

Adaptive Multi-Core Real- Time Scheduling Based on Reinforcement Learning

Orchestrated Co-scheduling, Resource Partitioning, and Power Capping on CPU-GPU Heterogeneous Systems via Machine Learning

Online Scheduling for Multi-Core Shared Reconfigurable Fabric

Cache-conscious off-line real-time scheduling for multi-core platforms: algorithms and implementation

A Cost-Optimized Data Parallel Task Scheduling in Multi-Core Resources Under Deadline and Budget Constraints

Efficient Multi-Processor Scheduling in Increasingly Realistic Models

SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators

Optimizing Task and Data Assignment on Multi-Core Systems with Multi-Port SPMs.

Effective GPU Sharing Under Compiler Guidance

Scheduling on Homogeneous Multi-Core System

Improving Scheduling Techniques in Heterogeneous Systems with Dynamic, On-Line Optimisations

Resource Scheduling Research of Polymorphism Task Based on Manycore Architecture

Lightweight ML-based Runtime Prefetcher Selection on Many-core Platforms

Optimizing resource utilization for large scale problems through architecture aware scheduling

Priority-Aware Near-Optimal Scheduling for Heterogeneous Multi-Core Systems with Specialized Accelerators

A Multi-Installment Scheduling Optimization Model Considering Processor Order

Comprehensive Scheduling Algorithm for Asymmetric Multi-Core Processors

Co-Optimizing Cache Partitioning and Multi-Core Task Scheduling: Exploit Cache Sensitivity or Not?