Abstract:Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. Simple job scheduling policies, such as Fair and FIFO scheduling, do not consider job sizes and may degrade the performance when jobs of varying sizes arrive. More elaborate job scheduling policies make the convenient assumption that jobs are recurring, and complete information about their sizes is available from their prior runs. In this paper, we design and implement an efficient and practical job scheduler for big data processing systems to achieve better performance even without prior information about job sizes. The superior performance of our job scheduler originates from the design of multiple level priority queues, where jobs are demoted to lower priority queues if the amount of service consumed so far reaches a certain threshold. In this case, jobs in need of a small amount of service can finish in the topmost several levels of queues, while jobs that need a large amount of service to complete are moved to lower priority queues to avoid head-of-line blocking. Our new job scheduler can effectively mimic the shortest job first scheduling policy without knowing the job sizes in advance. To demonstrate its performance, we have implemented our new job scheduler in YARN, a popular resource manager used by Hadoop/Spark, and validated its performance with experiments in both real testbeds including Amazon EC2 and large-scale trace-driven simulations. Our experimental and simulation results have strongly confirmed the effectiveness of our design: our new job scheduler can reduce the average job response time of the Fair scheduler by up to 45 percent and achieve better fairness at the same time.

Research of Hadoop Performance Optimization Based on the Scheduler

Performance optimization of computing task scheduling based on the Hadoop big data platform

An Adaptive Scheduling Algorithm for Heterogeneous Hadoop Systems

A Scheduler for Hadoop Based on Dynamic Resource Collection

Research of Task Scheduling Mechanism Based on Prediction of Memory Utilization

Research On Scheduling Scheme For Hadoop Clusters

RESEARCH ON MAPREDUCE SCHEDULING ALGORITHM ON HADOOP PLATFORM

Matchmaking: A New MapReduce Scheduling Technique

ACO Algorithm-based Parallel Job Scheduling Investigation on Hadoop

An Efficiency-Aware Scheduling for Data-Intensive Computations on Mapreduce Clusters

Research on job scheduling algorithm in Hadoop

A Load Balancing Algorithm Based on Fair Scheduler

OFScheduler: A Dynamic Network Optimizer for MapReduce in Heterogeneous Cluster

Study on Adaptive Delay Schedule Algorithm Based on Progress Control of Hadoop

An Efficient Job Scheduling for Map Reduce Clusters

A Load-Driven Task Scheduler with Adaptive DSC for MapReduce

Performance Comparison and Analysis of Yarn's Schedulers with Stress Cases

Research on hadoop greedy scheduler based on the fair

New Scheduling Algorithms for Improving Performance and Resource Utilization in Hadoop YARN Clusters

Low Latency Big Data Processing Without Prior Information

Research on Elastic Job Scheduling Model and Algorithm of Cloud Computing Based on Hadoop