Abstract:Distributed computing applications like MapReduce transfer massive amount of data between their successive processing stages. These data transfers, such as common shuffle and incast communication patterns, contribute most of the network traffic and thus have severe impacts on application performances. Despite such impacts, there has been relatively little work on decreasing the amount of traffic for computing such data transfers. We observe that the massive data flows in such a transfer already apply aggregate functions at the receiver side and the reduction in size between the input data and the output data is even pronounced. This motivates us to bring opportunities for performing interflow data aggregation during the transmission phase as early as possible rather than just at the receiver side. To this end, we first demonstrate the gain and feasibility of inter-flow data aggregation for data transfers in data centers with novel network structures. To achieve such a gain, such data transfers are normalized as the incast transfer. It is modeled as an incast minimal tree problem that is proved to be NP-hard in representative BCube and FBFLY data centers. We propose two approximate methods, the RS-based and ARS-based incast tree building methods, to generate an efficient incast tree based on only the labels of all incast members and the data center topology. We further present incremental methods to tackle the dynamic and fault-tolerant issues of the incast tree. Using a prototype implementation and large-scale simulations, we demonstrate that our method can significantly decrease the amount of network traffic, save the data center resources, and reduce the delay of the entire process of a job. Moreover, our proposals for BCube and FBFLY can also be applied to other novel data centers after minimal modifications.

Lessen Interflow Interference Using Virtual Channels Partitioning.

A Virtual Channel Allocation Algorithm for NoC.

Freeway: Adaptively Isolating the Elephant and Mice Flows on Different Transmission Paths

CQRD: A switch-based approach to flow interference in Data Center Networks

Congestion integrated control in virtualized clouds

A Dynamically-Allocated Virtual Channel Architecture with Congestion Awareness for On-Chip Routers

Escaping from Blocking: A Dynamic Virtual Channel for Pipelined Routers

RepFlow: Minimizing flow completion times with replicated flows in data centers

Alleviating Flow Interference in Data Center Networks Through Fine-Grained Switch Queue Management.

Interference-Aware Channel Assignment for On-demand Routing Protocols in Multi-radio Multi-channel Wireless Mesh Networks

VirtualKnotter: Online Virtual Machine Shuffling for Congestion Resolving in Virtualized Datacenter

Reducing the Southbound Interface Overhead for OpenFlow Based on the Flow Volume Characteristics

Isolating Mice and Elephant in Data Centers

Coupling Congestion Control and Flow Pausing in Data Center Network

An Efficient and Transparent Approach for Adaptive Intra-and Inter-Node Virtual Machine Communication in Virtualized Clouds.

Halflife

A Flit-Level Speedup Scheme for Network-on-chips Using Self-Reconfigurable Bi-Directional Channels

Flash: Joint Flow Scheduling and Congestion Control in Data Center Networks

A Low-Latency Virtual-Channel Router with Optimized Pipeline Stages for On-Chip Network

Swallow: Joint Online Scheduling and Coflow Compression in Datacenter Networks

Managing and Aggregating Data Transfers in Data Centers