Abstract:Over the last decade, we have witnessed growing data volumes generated and stored across geographically distributed datacenters. Processing such geo-distributed datasets may suffer from significant slowdown as the underlying network flows have to go through the inter-datacenter networks with relatively low and highly heterogeneous available link bandwidth. Thus, optimizing the transmissions of inter-datacenter flows, especially coflows that capture application-level semantics, is important for improving the communication performance of such geo-distributed applications. However, prior solutions on coflow scheduling have significant limitations: they schedule coflows with already-fixed endpoints of flows, making them insufficient to optimize the coflow completion time (CCT). In this article, we focus on the problem of jointly considering endpoint placement and coflow scheduling to minimize the average CCT of coflows across geo-distributed datacenters. To solve this problem without any prior knowledge of coflow arrivals, we present a coflow-aware optimization framework called SmartCoflow. In SmartCoflow, we first apply an approximate algorithm to obtain the endpoint placement and scheduling decisions for a single coflow. Based on the single-coflow solution, we then develop an efficient online algorithm to handle the dynamically arrived coflows. Through rigorous theoretical analysis, we prove that SmartCoflow has a non-trivial competitive ratio. We also extend SmartCoflow to incorporate various design choices or requirements of applications and operators, such as enforcing an inter-datacenter bandwidth usage budget and considering coflow deadline. Through experimental results from testbed implementation and trace-driven simulations, we demonstrate that SmartCoflow can reduce the average CCT, lower bandwidth usage, and improve coflow deadline meet rate, when compared t- the state-of-the-art scheduling-only method.

Freeway: Adaptively Isolating the Elephant and Mice Flows on Different Transmission Paths

Mice flow transmission mechanism: Delay-sensitive Fast-Pacing TCP

RFT: Toward Highly Reliable Flow Data Transmission in Network Measurement

Isolating Mice and Elephant in Data Centers

Flow-level and Efficient Traffic Engineering in Conventional Routing Systems

Don't be fat: Towards efficient online flow scheduling in data center networks

An Efficient Flow Detection and Scheduling Method in Data Center Networks

Key Flow First Prioritized Flow Scheduling Strategy In Multi-Tenant Data Centers

Cutting Long-Tail Latency of Routing Response in Software Defined Networks.

Flow-Level Traffic Scheduling in Data Center Networks

Flow-level Traffic Engineering in Conventional Networks with Hop-by-Hop Routing

Efficient Scheduling of Weighted Coflows in Data Centers

DRR-SFF: A Practical Scheduling Algorithm to Improve the Performance of Short Flows

Efficient Data Center Flow Scheduling Without Starvation Using Expansion Ratio

Endpoint-Flexible Coflow Scheduling Across Geo-Distributed Datacenters

Flow Optimization at Inter-Datacenter Networks for Application Run-time Acceleration

Rethinking Transfer Optimization in a Datacenter: Integrating Load Balancing with Multipath Flow Control.

Rapier: Integrating routing and scheduling for coflow-aware data center networks

FlowTele

Optimizing Flow Completion Time Via Adaptive Buffer Management in Data Center Networks

ALFE: A replacement policy to cache elephant flows in the presence of mice flooding