Abstract:In the rapidly expanding field of parallel processing, job schedulers are the "operating systems" of modern big data architectures and supercomputing systems. Job schedulers allocate computing resources and control the execution of processes on those resources. Historically, job schedulers were the domain of supercomputers, and job schedulers were designed to run massive, long-running computations over days and weeks. More recently, big data workloads have created a need for a new class of computations consisting of many short computations taking seconds or minutes that process enormous quantities of data. For both supercomputers and big data systems, the efficiency of the job scheduler represents a fundamental limit on the efficiency of the system. Detailed measurement and modeling of the performance of schedulers are critical for maximizing the performance of a large-scale computing system. This paper presents a detailed feature analysis of 15 supercomputing and big data schedulers. For big data workloads, the scheduler latency is the most important performance characteristic of the scheduler. A theoretical model of the latency of these schedulers is developed and used to design experiments targeted at measuring scheduler latency. Detailed benchmarking of four of the most popular schedulers (Slurm, Son of Grid Engine, Mesos, and Hadoop YARN) are conducted. The theoretical model is compared with data and demonstrates that scheduler performance can be characterized by two key parameters: the marginal latency of the scheduler $t_s$ and a nonlinear exponent $\alpha_s$. For all four schedulers, the utilization of the computing system decreases to < 10\% for computations lasting only a few seconds. Multilevel schedulers that transparently aggregate short computations can improve utilization for these short computations to > 90\% for all four of the schedulers that were tested.

An Adaptive Meta-Scheduler for Data-Intensive Applications

Application-adaptive Resource Scheduling in a Computational Grid

Performance-Driven Task and Data Co-scheduling Algorithms for Data-Intensive Applications in Grid Computing

A Deadline And Budget Constrained Cost-Time Optimization Algorithm For Scheduling Dependent Tasks In Grid Computing

Energy-aware application scheduling based on genetic algorithm.

Scalable System Scheduling for HPC and Big Data

Comparative Analysis of Energy-Efficient Scheduling Algorithms for Big Data Applications.

A Scheduler for Local Grid Resource Domain in High Throughput Environment

An Online Scheduling Algorithm for Assigning Jobs in the Computational Grid

Enhanced Adaptive Scheduling for the Grid Harvest Service

An Integrated Resource Management and Scheduling System for Grid Data Streaming Applications

Job Scheduling in High Performance Computing

Performance Modeling and Analysis for Resource Scheduling in Data Grids.

An Efficiency-Aware Scheduling for Data-Intensive Computations on Mapreduce Clusters

An Open Job Scheduling Service for Large-Scale Data Processing

Interconnection Network Energy-Aware Workflow Scheduling Algorithm on Heterogeneous Systems

Adaptive Scheduling Framework of Streaming Applications based on Resource Demand Prediction with Hybrid Algorithms

A License-aware Meta-Scheduling Engine in Grid Environment

Scheduling in Data Intensive and Network Aware (DIANA) Grid Environments

Scalable Hierarchical Scheduling for Multiprocessor Systems Using Adaptive Feedback-Driven Policies

Computing Field Scheduling: A Fully Decentralized Scheduling Approach for Grid Computing.