Abstract:Collecting flow records is a common practice of network operators and researchers for monitoring, diagnosing and understanding a network. Traditional tools like NetFlow face great challenges when both the speed and the complexity of the network traffic increase. To keep pace up, we propose HashFlow, a tool for more efficient and accurate collection and analysis of flow records. The central idea of HashFlow is to maintain accurate records for elephant flows, but summarized records for mice flows, by applying a novel collision resolution and record promotion strategy to hash tables. The performance bound can be analyzed with a probabilistic model, and with this strategy, HashFlow achieves a better utilization of space, and also more accurate flow records, without bringing extra complexity. We have implemented HashFlow, as well as several latest flow measurement algorithms such as FlowRadar, HashPipe and ElasticSketch, in a P4 software switch. Then we use traces from different operational networks to evaluate them. In these experiments, for various types of traffic analysis applications, HashFlow consistently demonstrates a clearly better performance against its state-of-the-art competitors. For example, using a small memory of 1 MB, HashFlow can accurately record around 55K flows, which is often 12.5% higher than the others. For estimating the sizes of 50K flows, HashFlow achieves a relative error of around 11.6%, while the estimation error of the best competitor is 42.9% higher. It detects 96.1% of the heavy hitters out of 250K flows with a size estimation error of 5.6%, which is 11.3% and 73.7% better than the best competitor respectively. At last, we show these merits of HashFlow come with almost no degradation of throughput.

Domain-Specific Framework for Performance Analysis

Graph-Centric Performance Analysis for Large-Scale Parallel Applications

Scaler: Efficient and Effective Cross Flow Analysis

BigDataflow: A Distributed Interprocedural Dataflow Analysis Framework

HPC System Software Enhanced by Source Code Analysis

An Efficient Task-Parallel Pipeline Programming Framework.

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

Evaluation and Analysis of Distributed Graph-Parallel Processing Frameworks

Analyzing Performance-Aware Code Changes in Software Development Process

Pipeflow: an Efficient Task-Parallel Pipeline Programming Framework Using Modern C++.

FlowMon-DPDK: Parsimonious Per-Flow Software Monitoring at Line Rate

An Integration of Several Different Data Flow Analysis Techniques

Parallelizing Flow-Sensitive Demand-Driven Points-to Analysis

FastFlow: Efficient Scalable Model-Driven Framework for Processing Massive Mobile Stream Data.

Understanding Performance Problems in Deep Learning Systems.

Taskflow: A General-Purpose Parallel and Heterogeneous Task Programming System

A Detailed and Real-Time Performance Monitoring Framework for Blockchain Systems.

Communication Analysis And Performance Prediction Of Parallel Applications On Large-Scale Machines

HashFlow For Better Flow Record Collection

PerfGen: Automated Performance Benchmark Generation for Big Data Analytics

Modeling and Analyzing the Performance of High-Speed Packet I/O