Abstract:To support higher demand for datacenter networks, networking researchers have proposed hybrid electrical/optical datacenter networks (Hybrid-DCN) that leverages optical circuit switching (OCS) along with traditional electrical packet switching (EPS). However, due to the high reconfiguration delay of OCS, OCS is used only for bulk data transfers between racks to amortize the reconfiguration delay. Existing job schedulers for data-parallel frameworks are not designed for Hybrid-DCN, since they neither place tasks to aggregate data traffic to take advantage of OCS, nor schedule tasks to minimize the Coflow completion time (CCT). In this paper, we describe the mismatch between existing job schedulers and the advanced Hybrid-DCN, introduce the requirements for the new scheduler, and present the implementation of Co-scheduler, a job scheduler for data-parallel frameworks that aims to improve job performance by placing the tasks of jobs to aggregate enough data traffic to better leverage OCS to minimize the CCT in Hybrid-DCN. Specifically, for every job, Co-scheduler computes guidelines on how many racks to place the job's input data and the job's tasks. The guidelines are dynamically generated based on the real-time job characteristics or predictable job characteristics from prior runs, with the aim of leveraging OCS whenever possible and efficient and minimizing CCT of jobs. Co-scheduler then schedules the tasks of jobs based on the guidelines. We evaluate the effectiveness of Co-scheduler using trace-driven simulation. The evaluation demonstrates that Co-scheduler can improve makespan, average job completion time, and average CCT of a workload by up to 56%, 61%, and 79%, respectively, compared to the state-of-the-art schedulers.

Data-Oriented Task Scheduling in Fixed- and Flexible-Grid Multilayer Inter-DC Optical Networks: A Comparison Study

On Establishing and Task Scheduling of Data-Oriented Vnf-Scs in an Optical DCI

Data Center Network Placement and Service Protection in All-Optical Mesh Networks

Interruptible Scheduling of Partially Re-configurable Optical Switching in Data Center Networks

Joint Scheduling and Routing for QoS Guaranteed Packet Transmission in Energy Efficient Reconfigurable WDM Mesh Networks

Adaptive Task Scheduling on Optical Grid

Scalable Data Center Network with distributed placement of component sets in optical networks

Efficient Coflow Scheduling in Hybrid-Switched Data Center Networks

Data Center Network Flow Scheduling Based on D PSO Algorithm

Task Scheduling and Lightpath Establishment in Optical Grids.

Differentiated Qos Guarantee In Optical Switches For Datacenters

Adaptive scheduling method available on optical grid

NegotiaToR: Towards A Simple Yet Effective On-demand Reconfigurable Datacenter Network

Joint Scheduling of Computation and Network Resource in Optical Grid

Effective *-Flow Schedule for Optical Circuit Switching Based Data Center Networks: A Comprehensive Survey

Co-Scheduler: A Coflow-Aware Data-Parallel Job Scheduler in Hybrid Electrical/Optical Datacenter Networks

Hybrid Packet/Fine-Grained-Optical Switching Architectures for Future Large-Scale Data Center Networks

Approximate Multicast Coflow Scheduling in Reconfigurable Data Center Networks.

Integrating Coflow and Circuit Scheduling for Optical Networks

Delay-Differentiated Scheduling in Optical Packet Switches for Cloud Data Centers

Online job scheduling for distributed machine learning in optical circuit switch networks