Abstract:As user-generated data diversifies, an increasing number of big data tasks now encompass unstructured data forms, including audio, imagery, and video content. The processing of these data often requires the support of GPU devices in the cluster. However, most of the existing cluster resource management frameworks lack effective decoupling of CPU resources and GPU resources, and the management granularity of GPU resources is too coarse, resulting in poor GPU sharing and low resource utilization. Given, for the inadequacy of the existing cluster resource management framework that cannot support batch stream integration and CPU-GPU resource scheduling, we design and implement a Hybrid Heterogeneous Resource Management (H-HRM) for CPU-GPU heterogeneous clusters. The resource queue binding mechanism on computing nodes provides flexible binding of CPU resources to GPU resources, which solves the problem that CPU resources and GPU resources in the cluster are difficult to decouple. According to the different uses of CPU or GPU as the main resource, a Hybrid Domain Resource Fairness (HDRF) model is proposed to realize the reasonable allocation of CPU resources and GPU resources. Through the queue stacking technology and the mechanism that multiple executors can run simultaneously on the queue, fine-grained sharing of GPU resources is realized and the utilization rate of GPU resources is improved. Finally, this paper realizes the docking of H-HRM and Spark programming framework and conducts real load tests, and after comparing with the performance of Mesos, it is proved that H-HRM can handle the mixed scenarios of batch jobs and stream processing jobs in the cluster, and the HDRF algorithm also greatly improves the utilization of the GPU.

Multiple Resources Scheduling for Diverse Workloads in Heterogeneous Datacenter

A Fair Scheduling Algorithm for Adaptive Heterogeneous Resources in Data Centers

An Efficient Fault-Tolerant Scheduling Algorithm for Periodic Real-Time Tasks in Heterogeneous Platforms

Performance-Driven Task and Data Co-scheduling Algorithms for Data-Intensive Applications in Grid Computing

Low Complexity Hierarchical Scheduling for Diverse Datacenter Jobs.

Smart-DRS: A Strategy of Dynamic Resource Scheduling in Cloud Data Center

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds.

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds

New Scheduling Algorithms for Improving Performance and Resource Utilization in Hadoop YARN Clusters

Multi-Tiered On-Demand Resource Scheduling for VM-Based Data Center

A Resource Co-Allocation Method for Load-Balance Scheduling over Big Data Platforms

Design and realization of hybrid resource management system for heterogeneous cluster

Coflow Scheduling in the Multi-Resource Environment

Energy-Efficient and Thermal-Aware Resource Management for Heterogeneous Datacenters

Multi-policy-aware MapReduce Resource Allocation and Scheduling for Smart Computing Cluster.

Scheduling Distributed Resources in Heterogeneous Private Clouds

Research on the Framework and Resource Scheduling Mechanisms of Hadoop YARN

DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms

H2O-Cloud: A Resource and Quality of Service-Aware Task Scheduling Framework for Warehouse-Scale Data Centers -- A Hierarchical Hybrid DRL (Deep Reinforcement Learning) based Approach

Energy Aware Task Scheduling in Data Centers.

Dynamic Multiworkflow Deadline and Budget Constrained Scheduling in Heterogeneous Distributed Systems