Abstract:The emerging deployment of geographically distributed data centers (DCs) incurs a significant amount of data transfers over the Internet. Such transfers are typically charged by Internet service providers with the widely adopted qth percentile charging model. In such a charging model, the time slots with top (100 - q) percent of data transmission do not affect the total transmission cost and can be viewed as "free." This brings the opportunity to optimize the scheduling of interDC transfers to minimize the entire transmission cost. However, a very little work has been done to exploit those "free" time slots for scheduling inter-DC transfers. The crux is that existing work either lacks a mechanism to accumulate traffic to "free" time slots, or inevitably relies on prior knowledge of future traffic arrival patterns. In this paper, we present TrafficShaper, a new scheduler that shapes the inter-DC traffic to exploit the "free" time slots involved in the qth percentile charging model, so as to reduce or even minimize the transmission cost. When shaping traffic, TrafficShaper advocates a simple principle: more traffic peaks should be scheduled in "free" time slots, while less traffic differentiation should be maintained among the remaining time slots. To this end, TrafficShaper designs a pricing-aware control framework, which makes online decisions for inter-DC transfers without requiring a prior knowledge of traffic arrivals. To verify the performance of TrafficShaper, we conduct rigorous theoretical analysis based on Lyapunov optimization techniques, large-scale trace-driven simulations, and small-scale testbed implementation. Results from rigorous mathematical analyses demonstrate that TrafficShaper can make the transmission cost arbitrarily close to the optimum value. Extensive trace-driven simulation results show that TrafficShaper can reduce the transmission cost by up to 40.23%, compared with the state-of-the-art solutions. The testbed experiments further verify that TrafficShaper can realistically reduce the transmission cost by up to 19.38%.

Templating Shuffles

Efficient. Scalable and Robust Data Shuffle Service for Distributed MapReduce Computing on Cloud

A Locality-Aware Shuffle Optimization on Fat-Tree Data Centers.

Ishuffle: Improving Hadoop Performance with Shuffle-on-Write

Exoshuffle: An Extensible Shuffle Architecture

Shadow: Exploiting the Power of Choice for Efficient Shuffling in MapReduce.

Phase-Reconfigurable Shuffle Optimization for Hadoop MapReduce

Efficient shuffle management with SCache for DAG computing frameworks.

EC-Shuffle: Dynamic Erasure Coding Optimization for Efficient and Reliable Shuffle in Spark

Efficient Shuffle Management for DAG Computing Frameworks Based on the FRQ Model

Exploiting Efficient and Scalable Shuffle Transfers in Future Data Center Networks

SmartShuffle: Managing Online Virtual Machine Shuffle in Virtualized Data Centers

Adaptive Preshuffling In Hadoop Clusters

Similarity-Based Node Distance Exploring and Locality-Aware Shuffle Optimization for Hadoop MapReduce

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Scalable, Fast Cloud Computing with Execution Templates

Rendering Differential Performance Preference Through Intelligent Network Edge in Cloud Data Centers

Real-Time Optimization of Dynamic Speed Scaling for Distributed Data Centers

Length Shuffle: Achieving High Performance and Flexibility for Data Center Networks Design.

TrafficShaper: Shaping Inter-Datacenter Traffic to Reduce the Transmission Cost.

OPS: Optimized Shuffle Management System for Apache Spark.