Abstract:In the rapidly expanding field of parallel processing, job schedulers are the "operating systems" of modern big data architectures and supercomputing systems. Job schedulers allocate computing resources and control the execution of processes on those resources. Historically, job schedulers were the domain of supercomputers, and job schedulers were designed to run massive, long-running computations over days and weeks. More recently, big data workloads have created a need for a new class of computations consisting of many short computations taking seconds or minutes that process enormous quantities of data. For both supercomputers and big data systems, the efficiency of the job scheduler represents a fundamental limit on the efficiency of the system. Detailed measurement and modeling of the performance of schedulers are critical for maximizing the performance of a large-scale computing system. This paper presents a detailed feature analysis of 15 supercomputing and big data schedulers. For big data workloads, the scheduler latency is the most important performance characteristic of the scheduler. A theoretical model of the latency of these schedulers is developed and used to design experiments targeted at measuring scheduler latency. Detailed benchmarking of four of the most popular schedulers (Slurm, Son of Grid Engine, Mesos, and Hadoop YARN) are conducted. The theoretical model is compared with data and demonstrates that scheduler performance can be characterized by two key parameters: the marginal latency of the scheduler $t_s$ and a nonlinear exponent $\alpha_s$. For all four schedulers, the utilization of the computing system decreases to < 10\% for computations lasting only a few seconds. Multilevel schedulers that transparently aggregate short computations can improve utilization for these short computations to > 90\% for all four of the schedulers that were tested.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in high - performance computing (HPC) and big data systems, the efficiency of the job scheduler fundamentally limits the performance of the entire system. Specifically, the paper focuses on how to optimize the performance of the scheduler through detailed measurement and modeling, especially when dealing with a large number of short - running computational tasks. These tasks usually need to be completed within a few seconds to a few minutes and deal with a huge amount of data. For this type of task, the latency of the scheduler (i.e., the time interval from task submission to start of execution) becomes a key indicator for measuring the performance of the scheduler. The main contributions of the paper include: 1. Classifying schedulers into different scheduler families and classifying a series of HPC and big data schedulers. 2. Listing the scheduler characteristics that are important for high - performance data analysis jobs and comparing the support of these characteristics by representative HPC and big data schedulers. 3. Pointing out that many characteristics are supported in different schedulers, but a key scheduler characteristic that has not been explored in detail is the latency of the job scheduler. 4. Deriving a mathematical description of the job start - up latency. 5. Proposing a benchmarking strategy for measuring the job start - up latency and benchmarking a set of representative HPC and big data schedulers. 6. Measuring the marginal latency and non - linear index of representative HPC and big data schedulers in the benchmarking results. Through detailed benchmarking of the four most popular schedulers (Slurm, Son of Grid Engine, Mesos, and Hadoop YARN), the paper found that for computations lasting only a few seconds, the system utilization of all four schedulers dropped below 10%. Multilevel schedulers (such as LLMapReduce) can increase the utilization of these four test schedulers to over 90% by transparently aggregating short - time computations. These findings are of great significance for designing and optimizing schedulers in future big data and HPC systems.

Scalable System Scheduling for HPC and Big Data

Time-sharing Parallel Applications Through Performance-Targeted Feedback-Controlled Real-Time Scheduling.

Job Scheduling in High Performance Computing

Low Latency Big Data Processing Without Prior Information

A Survey of System Scheduling for HPC and Big Data

Job Scheduling Without Prior Information in Big Data Processing Systems

A Simulator for Data-Intensive Job Scheduling

Job schedulers for Big data processing in Hadoop environment: testing real-life schedulers using benchmark programs

Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs

Exploring Plan-Based Scheduling for Large-Scale Computing Systems

A HPC Co-Scheduler with Reinforcement Learning

An Adaptive Meta-Scheduler for Data-Intensive Applications

Energy-efficient Deadline Scheduling for Heterogeneous Systems.

An Adaptive Scheduling Algorithm for Heterogeneous Hadoop Systems

Exploring the Relation Between Two Levels of Scheduling Using a Novel Simulation Approach

Performance evaluation of job schedulers on Hadoop YARN

PAS: Performance-Aware Job Scheduling for Big Data Processing Systems

Do the Hard Stuff First: Scheduling Dependent Computations in Data-Analytics Clusters

Impact of Workload and System Parameters on Next Generation Cluster Scheduling Mechanisms

A Survey of Scheduling Frameworks in Big Data Systems

Energy-efficient Task Scheduling on Heterogeneous Computing Systems by Linear Programming