Abstract:Over the last decade, we have witnessed growing data volumes generated and stored across geographically distributed datacenters. Processing such geo-distributed datasets may suffer from significant slowdown as the underlying network flows have to go through the inter-datacenter networks with relatively low and highly heterogeneous available link bandwidth. Thus, optimizing the transmissions of inter-datacenter flows, especially coflows that capture application-level semantics, is important for improving the communication performance of such geo-distributed applications. However, prior solutions on coflow scheduling have significant limitations: they schedule coflows with already-fixed endpoints of flows, making them insufficient to optimize the coflow completion time (CCT). In this article, we focus on the problem of jointly considering endpoint placement and coflow scheduling to minimize the average CCT of coflows across geo-distributed datacenters. To solve this problem without any prior knowledge of coflow arrivals, we present a coflow-aware optimization framework called SmartCoflow. In SmartCoflow, we first apply an approximate algorithm to obtain the endpoint placement and scheduling decisions for a single coflow. Based on the single-coflow solution, we then develop an efficient online algorithm to handle the dynamically arrived coflows. Through rigorous theoretical analysis, we prove that SmartCoflow has a non-trivial competitive ratio. We also extend SmartCoflow to incorporate various design choices or requirements of applications and operators, such as enforcing an inter-datacenter bandwidth usage budget and considering coflow deadline. Through experimental results from testbed implementation and trace-driven simulations, we demonstrate that SmartCoflow can reduce the average CCT, lower bandwidth usage, and improve coflow deadline meet rate, when compared t- the state-of-the-art scheduling-only method.

Using the Macroflow Abstraction to Minimize Machine Slot-time Spent on Networking in Hadoop

Macroflow: A Fine-Grained Networking Abstraction for Job Completion Time Oriented Scheduling in Datacenters

MINT: Empowering Multiple Flow Definition Query for Network-Wide Measurement.

Metaflow: A Better Traffic Abstraction for Distributed Applications.

RepFlow: Minimizing flow completion times with replicated flows in data centers

Single-machine scheduling to minimize total convex resource consumption with a constraint on total weighted flow time

MSS: Exploiting Mapping Score for CQF Start Time Planning in Time-Sensitive Networking

Minimizing Tardiness for Data-Intensive Applications in Heterogeneous Systems: A Matching Theory Perspective.

Fast Coflow Scheduling Via Traffic Compression and Stage Pipelining in Datacenter Networks

Metaflow: A DAG-Based Network Abstraction for Distributed Applications

Flash: Joint Flow Scheduling and Congestion Control in Data Center Networks

A Deadline-Aware Coflow Scheduling Approach for Big Data Applications.

Towards Minimal Tardiness Of Data-Intensive Applications In Heterogeneous Networks

Scheduling Coflows for Minimizing the Makespan in Identical Parallel Networks

Freeway: Adaptively Isolating the Elephant and Mice Flows on Different Transmission Paths

Endpoint-Flexible Coflow Scheduling Across Geo-Distributed Datacenters

Minimizing Datacenter Flow Completion Times with Server-Based Flow Scheduling

Scheduling Dependent Coflows to Minimize the Total Weighted Job Completion Time in Datacenters.

Key Flow First Prioritized Flow Scheduling Strategy In Multi-Tenant Data Centers

Joint Online Coflow Routing and Scheduling in Data Center Networks

Scheduling Coflows for Minimizing the Total Weighted Completion Time in Heterogeneous Parallel Networks