A Near Optimal Multi-Faced Job Scheduler For Datacenter Workloads

Hengky Susanto,Ahmed M. Abdelmoniem,Honggang Zhang,Benyuan Liu,Don Towsley

DOI: https://doi.org/10.1109/ICDCS.2019.00200

2019-01-01

Abstract:As data-parallel applications process more complex data, the dependencies between computation jobs in a multi-stage job also becomes more complicated. However, most of the existing scheduling solutions primarily rely on total bytes sent (job size) to differentiate jobs where jobs with less bytes sent are prioritized over the larger ones. This approach overlooks the fact that jobs may consist of multiple computation stages, and that the completion of a computation job stage depends on the completion of other jobs' stage. In this paper, we present a coflow scheduler of multi-stage jobs that minimizes the average job completion time. Our solution prioritizes jobs based on the multi-faceted characteristics of multi-stage job structure per stage, instead of total bytes sent. Our experiments show that our approach provides twice the performance of existing solutions on average and by four times in bursty traffic scenario.