Abstract:To support higher demand for datacenter networks, networking researchers have proposed hybrid electrical/optical datacenter networks (Hybrid-DCN) that leverages optical circuit switching (OCS) along with traditional electrical packet switching (EPS). However, due to the high reconfiguration delay of OCS, OCS is used only for bulk data transfers between racks to amortize the reconfiguration delay. Existing job schedulers for data-parallel frameworks are not designed for Hybrid-DCN, since they neither place tasks to aggregate data traffic to take advantage of OCS, nor schedule tasks to minimize the Coflow completion time (CCT). In this paper, we describe the mismatch between existing job schedulers and the advanced Hybrid-DCN, introduce the requirements for the new scheduler, and present the implementation of Co-scheduler, a job scheduler for data-parallel frameworks that aims to improve job performance by placing the tasks of jobs to aggregate enough data traffic to better leverage OCS to minimize the CCT in Hybrid-DCN. Specifically, for every job, Co-scheduler computes guidelines on how many racks to place the job's input data and the job's tasks. The guidelines are dynamically generated based on the real-time job characteristics or predictable job characteristics from prior runs, with the aim of leveraging OCS whenever possible and efficient and minimizing CCT of jobs. Co-scheduler then schedules the tasks of jobs based on the guidelines. We evaluate the effectiveness of Co-scheduler using trace-driven simulation. The evaluation demonstrates that Co-scheduler can improve makespan, average job completion time, and average CCT of a workload by up to 56%, 61%, and 79%, respectively, compared to the state-of-the-art schedulers.

BAS : Branch-Aware Scheduling for Iterative Data-Parallel Jobs

A Deadline-Aware Coflow Scheduling Approach for Big Data Applications.

Efficient Scheduling for Multi-Stage Coflows

Distributed Bottleneck-Aware Coflow Scheduling in Data Centers

Towards Practical and Near-Optimal Coflow Scheduling for Data Center Networks

A Branch and Bound Algorithm for Scheduling Jobs with Controllable Processing Times on a Single Machine to Meet Due Dates.

Co-Scheduler: A Coflow-Aware Data-Parallel Job Scheduler in Hybrid Electrical/Optical Datacenter Networks

A Near Optimal Multi-Faced Job Scheduler For Datacenter Workloads

Efficient Resource Constrained Scheduling Using Parallel Two-Phase Branch-and-Bound Heuristics

Efficient Coflow Scheduling of Multi-Stage Jobs with Isolation Guarantee

Scheduling Dependent Coflows to Minimize the Total Weighted Job Completion Time in Datacenters.

Coflow Scheduling in Data Centers: Routing and Bandwidth Allocation

Minimizing Average Coflow Completion Time with Decentralized Scheduling.

Scheduling Coflows of Multi-Stage Jobs under Network Resource Constraints.

Fast Coflow Scheduling Via Traffic Compression and Stage Pipelining in Datacenter Networks

Coflow Scheduling in Input-Queued Switches: Optimal Delay Scaling and Algorithms

Building Real-Time Parallel Task Systems on Multi-Cores: A Hierarchical Scheduling Approach.

Seagull -- A Real-Time Coflow Scheduling System

Runtime Parallel Incremental Scheduling of DAGs

Macroflow: A Fine-Grained Networking Abstraction for Job Completion Time Oriented Scheduling in Datacenters

On Multiprocessor System Scheduling