Abstract:To support higher demand for datacenter networks, networking researchers have proposed hybrid electrical/optical datacenter networks (Hybrid-DCN) that leverages optical circuit switching (OCS) along with traditional electrical packet switching (EPS). However, due to the high reconfiguration delay of OCS, OCS is used only for bulk data transfers between racks to amortize the reconfiguration delay. Existing job schedulers for data-parallel frameworks are not designed for Hybrid-DCN, since they neither place tasks to aggregate data traffic to take advantage of OCS, nor schedule tasks to minimize the Coflow completion time (CCT). In this paper, we describe the mismatch between existing job schedulers and the advanced Hybrid-DCN, introduce the requirements for the new scheduler, and present the implementation of Co-scheduler, a job scheduler for data-parallel frameworks that aims to improve job performance by placing the tasks of jobs to aggregate enough data traffic to better leverage OCS to minimize the CCT in Hybrid-DCN. Specifically, for every job, Co-scheduler computes guidelines on how many racks to place the job's input data and the job's tasks. The guidelines are dynamically generated based on the real-time job characteristics or predictable job characteristics from prior runs, with the aim of leveraging OCS whenever possible and efficient and minimizing CCT of jobs. Co-scheduler then schedules the tasks of jobs based on the guidelines. We evaluate the effectiveness of Co-scheduler using trace-driven simulation. The evaluation demonstrates that Co-scheduler can improve makespan, average job completion time, and average CCT of a workload by up to 56%, 61%, and 79%, respectively, compared to the state-of-the-art schedulers.

Low Complexity Hierarchical Scheduling for Diverse Datacenter Jobs.

A Novel Job Scheduling Model to Enhance Efficiency and Overall User Fairness of Cloud Computing Environment.

Multiple Resources Scheduling for Diverse Workloads in Heterogeneous Datacenter

H2O-Cloud: A Resource and Quality of Service-Aware Task Scheduling Framework for Warehouse-Scale Data Centers -- A Hierarchical Hybrid DRL (Deep Reinforcement Learning) based Approach

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds.

An Integrated Dynamic Resource Scheduling Framework in On-Demand Clouds

A Near Optimal Multi-Faced Job Scheduler For Datacenter Workloads

Hierarchical scheduling for multi-composite tasks in cloud manufacturing

Smart-DRS: A Strategy of Dynamic Resource Scheduling in Cloud Data Center

A Fair Scheduling Algorithm for Adaptive Heterogeneous Resources in Data Centers

Cluster Scheduler on Heterogeneous Cloud

Coflow Scheduling in the Multi-Resource Environment

Highly Complex Resource Scheduling for Stochastic Demands in Heterogeneous Clouds

CloudCoaster: Transient-aware Bursty Datacenter Workload Scheduling

Scheduling Distributed Resources in Heterogeneous Private Clouds

Co-Scheduler: A Coflow-Aware Data-Parallel Job Scheduler in Hybrid Electrical/Optical Datacenter Networks

Multi-Tiered On-Demand Resource Scheduling for VM-Based Data Center

Preemptive and Low Latency Datacenter Scheduling via Lightweight Containers

Hierarchical Multi-resource Fair Queueing for Packet Processing

MQWAGS: Research on Job Scheduling Algorithms Based on Cloud Computing

A Cooperative Learning-Aware Dynamic Hierarchical Hyper-Heuristic for Distributed Heterogeneous Mixed No-Wait Flow-Shop Scheduling