Abstract:As user-generated data diversifies, an increasing number of big data tasks now encompass unstructured data forms, including audio, imagery, and video content. The processing of these data often requires the support of GPU devices in the cluster. However, most of the existing cluster resource management frameworks lack effective decoupling of CPU resources and GPU resources, and the management granularity of GPU resources is too coarse, resulting in poor GPU sharing and low resource utilization. Given, for the inadequacy of the existing cluster resource management framework that cannot support batch stream integration and CPU-GPU resource scheduling, we design and implement a Hybrid Heterogeneous Resource Management (H-HRM) for CPU-GPU heterogeneous clusters. The resource queue binding mechanism on computing nodes provides flexible binding of CPU resources to GPU resources, which solves the problem that CPU resources and GPU resources in the cluster are difficult to decouple. According to the different uses of CPU or GPU as the main resource, a Hybrid Domain Resource Fairness (HDRF) model is proposed to realize the reasonable allocation of CPU resources and GPU resources. Through the queue stacking technology and the mechanism that multiple executors can run simultaneously on the queue, fine-grained sharing of GPU resources is realized and the utilization rate of GPU resources is improved. Finally, this paper realizes the docking of H-HRM and Spark programming framework and conducts real load tests, and after comparing with the performance of Mesos, it is proved that H-HRM can handle the mixed scenarios of batch jobs and stream processing jobs in the cluster, and the HDRF algorithm also greatly improves the utilization of the GPU.

Prediction-Based and Locality-Aware Task Scheduling for Parallelizing Video Transcoding over Heterogeneous MapReduce Cluster

Parallelizing video transcoding using Map-Reduce-based cloud computing

Parallelizing Video Transcoding with Load Balancing on Cloud Computing

On-Line Scheduling of Parallel Jobs in Heterogeneous Multiple Clusters

A Dispatching-Rule-Based Task Scheduling Policy for MapReduce with Multi-type Jobs in Heterogeneous Environments

A Complexity Aware Scheduler With Dynamic Slot Allocation for Cloud Video Transcoding

Hybrid Coded MapReduce for Latency-Constrained Tasks with Straggling Servers

Hybrid Map Task Scheduling for GPU-Based Heterogeneous Clusters

Load scheduling for distributed edge computing: A communication-computation tradeoff

Optimizing Resource Allocation for Data-Parallel Jobs Via GCN-Based Prediction

Communication-Efficient Computation Load Scheduling for Delay-Constrained Services

Task Scheduling for GPU Heterogeneous Cluster.

ReLoca: Optimize Resource Allocation for Data-parallel Jobs Using Deep Learning

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Scheduling Algorithm Based on Prefetching in MapReduce Clusters

H-PFSP: Efficient Hybrid Parallel PFSP Protected Scheduling for MapReduce System

Design and Implementation of a Real-time Scheduling Algorithm for MapReduce

Task Scheduling Greedy Heuristics for GPU Heterogeneous Cluster Involving the Weights of the Processor

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Performance Analysis and Modeling of Video Transcoding Using Heterogeneous Cloud Services

Design and realization of hybrid resource management system for heterogeneous cluster