Abstract:To support higher demand for datacenter networks, networking researchers have proposed hybrid electrical/optical datacenter networks (Hybrid-DCN) that leverages optical circuit switching (OCS) along with traditional electrical packet switching (EPS). However, due to the high reconfiguration delay of OCS, OCS is used only for bulk data transfers between racks to amortize the reconfiguration delay. Existing job schedulers for data-parallel frameworks are not designed for Hybrid-DCN, since they neither place tasks to aggregate data traffic to take advantage of OCS, nor schedule tasks to minimize the Coflow completion time (CCT). In this paper, we describe the mismatch between existing job schedulers and the advanced Hybrid-DCN, introduce the requirements for the new scheduler, and present the implementation of Co-scheduler, a job scheduler for data-parallel frameworks that aims to improve job performance by placing the tasks of jobs to aggregate enough data traffic to better leverage OCS to minimize the CCT in Hybrid-DCN. Specifically, for every job, Co-scheduler computes guidelines on how many racks to place the job's input data and the job's tasks. The guidelines are dynamically generated based on the real-time job characteristics or predictable job characteristics from prior runs, with the aim of leveraging OCS whenever possible and efficient and minimizing CCT of jobs. Co-scheduler then schedules the tasks of jobs based on the guidelines. We evaluate the effectiveness of Co-scheduler using trace-driven simulation. The evaluation demonstrates that Co-scheduler can improve makespan, average job completion time, and average CCT of a workload by up to 56%, 61%, and 79%, respectively, compared to the state-of-the-art schedulers.

A Communication-Contention-Aware Privacy-Preserving Workflow Scheduling Method for Geo-Distributed Datacenters

Scheduling Workflows with Privacy Protection Constraints for Big Data Applications on Cloud.

Deadline-constrained cost-aware workflow scheduling in hybrid cloud

Cloud-edge-end workflow scheduling with multiple privacy levels

Graph partition–based data and task co‐scheduling of scientific workflow in geo‐distributed datacenters

Cost and Reliability Aware Scheduling of Workflows Across Multiple Clouds with Security Constraints

Co-Scheduler: A Coflow-Aware Data-Parallel Job Scheduler in Hybrid Electrical/Optical Datacenter Networks

Cost-Efficient Workflow Scheduling Algorithm for Applications With Deadline Constraint on Heterogeneous Clouds

A Deadline-Aware Coflow Scheduling Approach for Big Data Applications.

Cost-Efficient Task Scheduling for Geo-distributed Data Analytics.

Energy-Aware Cloud Workflow Applications Scheduling With Geo-Distributed Data

Cost and makespan scheduling of workflows in clouds using list multiobjective optimization technique

Big Data Processing Workflows Oriented Real-Time Scheduling Algorithm using Task-Duplication in Geo-Distributed Clouds

Fault-tolerant scheduling and data placement for scientific workflow processing in geo-distributed clouds

Workload-Aware Scheduling Across Geo-Distributed Data Centers

Hypergraph-partitioning-based online joint scheduling of tasks and data

Cost-Efficient Scheduling of Bulk Transfers in Inter-Datacenter WANs

Fairness scheduling with dynamic priority for multi workflow on heterogeneous systems

A Security and Cost Aware Scheduling Algorithm for Heterogeneous Tasks of Scientific Workflow in Clouds

PWSA3C: Prioritized Workflow Scheduler in Cloud Computing Using Asynchronous Advantage Actor Critic (A3C) Algorithm

Multi-objective Scheduling for Scientific Workflow in Multicloud Environment