Abstract:Traditional network resource management mechanisms are mainly flow or packet based. Recently, coflow has been proposed as a new abstraction to capture the communication patterns in a rich set of data parallel applications in data centers. Coflows effectively model the application-level semantics of network resource usage, so high-level optimization goals, such as reducing the transfer latency of applications, can be better achieved by taking coflows as the basic elements in network resource allocation or scheduling. Although efficient coflow scheduling methods have been studied, in this paper, we advocate to schedule weighted coflows as a further step in this direction, where weights are used to express the importances or priorities of different coflows or their corresponding applications. We propose the Weighted Coflow Completion Time (WCCT) minimization problem and a (2 - 2/n+1)-approximate optimal offline algorithm, where n is the concurrent number of coflows. We then design an information-agnostic online algorithm named IAOA to dynamically schedule coflows according to their weights and the instantaneous network condition. We also design and implement a coflow scheduling system named FlyTransfer, which can use the online algorithm as its scheduling method. We test the performance of FlyTransfer by trace-driven simulations as well as real deployment in openstack. Our evaluation results show that, compared to the latest information-agnostic coflow scheduling algorithms, FlyTransfer can reduce more than 40 percent of the WCCT, and more than 30 percent of the completion time for coflows with above-the-average level of importance. It even outperforms the most efficient clairvoyant coflow scheduling method by reducing around 30 percent WCCT, and 25-30 percent of the completion time for coflows with above-the-average importance, respectively.

Efficient Coflow Scheduling of Multi-Stage Jobs with Isolation Guarantee

Efficient Scheduling for Multi-Stage Coflows

Fast and Isolation Guaranteed Coflow Scheduling Via Traffic Forecasting in Multi-Tenant Environment

Scheduling Dependent Coflows to Minimize the Total Weighted Job Completion Time in Datacenters.

Scheduling Coflows of Multi-Stage Jobs under Network Resource Constraints.

Distributed Bottleneck-Aware Coflow Scheduling in Data Centers

Scheduling Dependent Coflows with Guaranteed Job Completion Time

Towards Practical and Near-Optimal Coflow Scheduling for Data Center Networks

Fair Coflow Scheduling via Controlled Slowdown

Coflow Scheduling in the Multi-Resource Environment

Efficient Scheduling of Weighted Coflows in Data Centers

Fast Coflow Scheduling Via Traffic Compression and Stage Pipelining in Datacenter Networks

Joint Online Coflow Routing and Scheduling in Data Center Networks

Coflow Scheduling in Data Centers: Routing and Bandwidth Allocation

Leveraging Endpoint Flexibility when Scheduling Coflows across Geo-distributed Datacenters.

RICH: Strategy-proof and Efficient Coflow Scheduling in Non-Cooperative Environments

Efficient Online Coflow Routing And Scheduling

Efficient and Fair: Information-Agnostic Online Coflow Scheduling by Combining Limited Multiplexing with DRL

Scheduling Coflows with Incomplete Information

Efficient Coflow Scheduling in Hybrid-Switched Data Center Networks

Seagull -- A Real-Time Coflow Scheduling System