Abstract:Traditional network resource management mechanisms are mainly flow or packet based. Recently, coflow has been proposed as a new abstraction to capture the communication patterns in a rich set of data parallel applications in data centers. Coflows effectively model the application-level semantics of network resource usage, so high-level optimization goals, such as reducing the transfer latency of applications, can be better achieved by taking coflows as the basic elements in network resource allocation or scheduling. Although efficient coflow scheduling methods have been studied, in this paper, we advocate to schedule weighted coflows as a further step in this direction, where weights are used to express the importances or priorities of different coflows or their corresponding applications. We propose the Weighted Coflow Completion Time (WCCT) minimization problem and a (2 - 2/n+1)-approximate optimal offline algorithm, where n is the concurrent number of coflows. We then design an information-agnostic online algorithm named IAOA to dynamically schedule coflows according to their weights and the instantaneous network condition. We also design and implement a coflow scheduling system named FlyTransfer, which can use the online algorithm as its scheduling method. We test the performance of FlyTransfer by trace-driven simulations as well as real deployment in openstack. Our evaluation results show that, compared to the latest information-agnostic coflow scheduling algorithms, FlyTransfer can reduce more than 40 percent of the WCCT, and more than 30 percent of the completion time for coflows with above-the-average level of importance. It even outperforms the most efficient clairvoyant coflow scheduling method by reducing around 30 percent WCCT, and 25-30 percent of the completion time for coflows with above-the-average importance, respectively.

Minimizing Datacenter Flow Completion Times with Server-Based Flow Scheduling

Towards Stable Flow Scheduling in Data Centers.

RepFlow: Minimizing flow completion times with replicated flows in data centers

LAFS: Learning-Based Application-Agnostic Flow Scheduling for Datacenters

Key Flow First Prioritized Flow Scheduling Strategy In Multi-Tenant Data Centers

HyLine: a Simple and Practical Flow Scheduling for Commodity Datacenters.

PIAS: Practical Information-Agnostic Flow Scheduling for Commodity Data Centers

START:Sensible Traffic Scheduling in Dynamic Data Center Networks

Don't be fat: Towards efficient online flow scheduling in data center networks

Flash: Joint Flow Scheduling and Congestion Control in Data Center Networks

Efficient Data Center Flow Scheduling Without Starvation Using Expansion Ratio

To schedule or not to schedule: when no-scheduling can beat the best-known flow scheduling algorithm in datacenter networks

An Efficient Flow Detection and Scheduling Method in Data Center Networks

One More Queue Is Enough: Minimizing Flow Completion Time With Explicit Priority Notification

Flow-Level Traffic Scheduling in Data Center Networks

Fast Coflow Scheduling Via Traffic Compression and Stage Pipelining in Datacenter Networks

Distributed Information-Agnostic Flow Scheduling in Data Centers Based on Wait-Time.

Efficient Scheduling of Weighted Coflows in Data Centers

Energy-Efficient Flow Scheduling and Routing with Hard Deadlines in Data Center Networks

CWND: A Coarse, but Simple, Efficient Metric to Reduce Short Flow Completion Time in Data Centers

DiFS: Distributed Flow Scheduling for Data Center Networks