Abstract:Over the last decade, we have witnessed growing data volumes generated and stored across geographically distributed datacenters. Processing such geo-distributed datasets may suffer from significant slowdown as the underlying network flows have to go through the inter-datacenter networks with relatively low and highly heterogeneous available link bandwidth. Thus, optimizing the transmissions of inter-datacenter flows, especially coflows that capture application-level semantics, is important for improving the communication performance of such geo-distributed applications. However, prior solutions on coflow scheduling have significant limitations: they schedule coflows with already-fixed endpoints of flows, making them insufficient to optimize the coflow completion time (CCT). In this article, we focus on the problem of jointly considering endpoint placement and coflow scheduling to minimize the average CCT of coflows across geo-distributed datacenters. To solve this problem without any prior knowledge of coflow arrivals, we present a coflow-aware optimization framework called SmartCoflow. In SmartCoflow, we first apply an approximate algorithm to obtain the endpoint placement and scheduling decisions for a single coflow. Based on the single-coflow solution, we then develop an efficient online algorithm to handle the dynamically arrived coflows. Through rigorous theoretical analysis, we prove that SmartCoflow has a non-trivial competitive ratio. We also extend SmartCoflow to incorporate various design choices or requirements of applications and operators, such as enforcing an inter-datacenter bandwidth usage budget and considering coflow deadline. Through experimental results from testbed implementation and trace-driven simulations, we demonstrate that SmartCoflow can reduce the average CCT, lower bandwidth usage, and improve coflow deadline meet rate, when compared t- the state-of-the-art scheduling-only method.

Joint Optimization of Task Placement and Routing in Minimizing Inter-Dc Coflow Completion Time

Joint optimization of task mapping and routing for service provisioning in distributed datacenters

Joint Online Coflow Routing and Scheduling in Data Center Networks

RPC: Joint Online Reducer Placement and Coflow Bandwidth Scheduling for Clusters

Joint Online Optimization of Task Rescheduling and Data Redistribution

Towards Practical and Near-Optimal Coflow Scheduling for Data Center Networks

Endpoint-Flexible Coflow Scheduling Across Geo-Distributed Datacenters

Optimizing the Cost-Performance Tradeoff for Coflows Across Geo-Distributed Datacenters

Efficient Online Coflow Routing And Scheduling

Coflow Scheduling in Data Centers: Routing and Bandwidth Allocation

Efficient Coflow Scheduling in Hybrid-Switched Data Center Networks

Joint Scheduling of Tasks and Network Flows in Big Data Clusters

Leveraging Endpoint Flexibility when Scheduling Coflows across Geo-distributed Datacenters.

Optimization of Joint Delivery Problem Based on Flow Control and Dynamic Programming Model

Multi-Hop Multi-Task Partial Computation Offloading in Collaborative Edge Computing

Scheduling Dependent Coflows to Minimize the Total Weighted Job Completion Time in Datacenters.

Leveraging Joint Allocation of Multidimensional Resources for Distributed Task Assignment

Joint VM Placement and Topology Optimization for Traffic Scalability in Dynamic Datacenter Networks.

Joint Optimization of VNF Placement and Flow Scheduling in Mobile Core Network

Greening Geo-distributed Data Centers by Joint Optimization of Request Routing and Virtual Machine Scheduling

VirtCo: joint coflow scheduling and virtual machine placement in cloud data centers