Abstract:In the rapidly expanding field of parallel processing, job schedulers are the "operating systems" of modern big data architectures and supercomputing systems. Job schedulers allocate computing resources and control the execution of processes on those resources. Historically, job schedulers were the domain of supercomputers, and job schedulers were designed to run massive, long-running computations over days and weeks. More recently, big data workloads have created a need for a new class of computations consisting of many short computations taking seconds or minutes that process enormous quantities of data. For both supercomputers and big data systems, the efficiency of the job scheduler represents a fundamental limit on the efficiency of the system. Detailed measurement and modeling of the performance of schedulers are critical for maximizing the performance of a large-scale computing system. This paper presents a detailed feature analysis of 15 supercomputing and big data schedulers. For big data workloads, the scheduler latency is the most important performance characteristic of the scheduler. A theoretical model of the latency of these schedulers is developed and used to design experiments targeted at measuring scheduler latency. Detailed benchmarking of four of the most popular schedulers (Slurm, Son of Grid Engine, Mesos, and Hadoop YARN) are conducted. The theoretical model is compared with data and demonstrates that scheduler performance can be characterized by two key parameters: the marginal latency of the scheduler $t_s$ and a nonlinear exponent $\alpha_s$. For all four schedulers, the utilization of the computing system decreases to < 10\% for computations lasting only a few seconds. Multilevel schedulers that transparently aggregate short computations can improve utilization for these short computations to > 90\% for all four of the schedulers that were tested.

A Survey of Scheduling Frameworks in Big Data Systems

Job Scheduling for Big Data Analytical Applications in Clouds: A Taxonomy Study

A taxonomy and survey on scheduling algorithms for scientific workflows in IaaS cloud computing environments

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds.

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds

A Survey on Mapreduce Scheduling in Cloud Computing

A Survey of System Scheduling for HPC and Big Data

A Novel Architecture Of Scheduling System For Big Data Framework

Scheduling in distributed systems: A cloud computing perspective

Survey of Hybrid Cloud Workflow Scheduling

Scientific workflow scheduling algorithms in cloud environments: a comprehensive taxonomy, survey, and future directions

Valuable survey on scheduling algorithms in the cloud with various publications

Typical Big Data Computing Frameworks

Scalable System Scheduling for HPC and Big Data

Analysis of Information Management and Scheduling Technology in Hadoop

Scalability Analysis of Request Scheduling in Cloud Computing

Job Scheduling Without Prior Information in Big Data Processing Systems

A Dynamical and Load-Balanced Flow Scheduling Approach for Big Data Centers in Clouds.

A Survey on Grid Scheduling Systems

Optimization of Big Data Parallel Scheduling Based on Dynamic Clustering Scheduling Algorithm