Abstract:Traditional network resource management mechanisms are mainly flow or packet based. Recently, coflow has been proposed as a new abstraction to capture the communication patterns in a rich set of data parallel applications in data centers. Coflows effectively model the application-level semantics of network resource usage, so high-level optimization goals, such as reducing the transfer latency of applications, can be better achieved by taking coflows as the basic elements in network resource allocation or scheduling. Although efficient coflow scheduling methods have been studied, in this paper, we advocate to schedule weighted coflows as a further step in this direction, where weights are used to express the importances or priorities of different coflows or their corresponding applications. We propose the Weighted Coflow Completion Time (WCCT) minimization problem and a (2 - 2/n+1)-approximate optimal offline algorithm, where n is the concurrent number of coflows. We then design an information-agnostic online algorithm named IAOA to dynamically schedule coflows according to their weights and the instantaneous network condition. We also design and implement a coflow scheduling system named FlyTransfer, which can use the online algorithm as its scheduling method. We test the performance of FlyTransfer by trace-driven simulations as well as real deployment in openstack. Our evaluation results show that, compared to the latest information-agnostic coflow scheduling algorithms, FlyTransfer can reduce more than 40 percent of the WCCT, and more than 30 percent of the completion time for coflows with above-the-average level of importance. It even outperforms the most efficient clairvoyant coflow scheduling method by reducing around 30 percent WCCT, and 25-30 percent of the completion time for coflows with above-the-average importance, respectively.

Achieving Efficient Scheduling Based on Accurate Measurement of Small Flows in Data Center

An effective and accurate flow size measurement using funnel-shaped sketch

Fast Coflow Scheduling Via Traffic Compression and Stage Pipelining in Datacenter Networks

Agc Sketch: an Effective and Accurate Per-Flow Measurement to Adapt Flow Size Distribution

Deadline-aware and Energy-Efficient Dynamic Flow Scheduling in Data Center Network.

Hierarchical Sketch: an Efficient Solution for Threshold-t Flows Measurement in High-Speed Networks

Per-Flow Network Measurement With Distributed Sketch

Efficient Scheduling for Multi-Stage Coflows

Key Flow First Prioritized Flow Scheduling Strategy In Multi-Tenant Data Centers

Priority-Aware Per-flow Measurement Using Cuckoo Sketch.

ChainSketch: an Efficient and Accurate Sketch for Heavy Flow Detection

Swallow: Joint Online Scheduling and Coflow Compression in Datacenter Networks

Minimizing Datacenter Flow Completion Times with Server-Based Flow Scheduling

Rate-aware Flow Scheduling for Commodity Data Center Networks.

Efficient Scheduling of Weighted Coflows in Data Centers

Pyramid Family: Generic Frameworks for Accurate and Fast Flow Size Measurement

START:Sensible Traffic Scheduling in Dynamic Data Center Networks

To schedule or not to schedule: when no-scheduling can beat the best-known flow scheduling algorithm in datacenter networks

Clustering-preserving Network Flow Sketching

Towards Stable Flow Scheduling in Data Centers.

Towards Practical and Near-Optimal Coflow Scheduling for Data Center Networks