Abstract:In high-speed networks, flow-level traffic measurement is an essential tool to understand how network bandwidth is consumed and support the detection of anomalous traffic. While many prior work focuses on tracking the frequent flows (a.k.a. heavy hitters), in this paper, we put more focus on tracking the persistent flows, which jointly consider the frequency, duration and regularity of the packet arrival events of a flow. Although this more generalized metric called persistence has been defined before, it is still unknown how to use limited memory on data plane to monitor the top-k persistent flows in a recent time interval. In this paper, we propose an algorithm named PFD-DW built upon cuckoo filter (an improved version of hash table with better memory efficiency), to monitor the top-k persistent flows under the time-decaying window. It can be regarded as a variant of cuckoo filter, which transforms each bucket into a bucket-level min-heap. Its advantage is that, when the table is full and a packet of a new flow arrives, it can select the least persistent flow along the cuckoo kicking path as the victim of flow replacement. We deliberately avoid the scanning of the entire table to keep the high time efficiency. Based on real-world network traffic traces, we evaluate the performance of our DAKP-CF, a degraded version of PFD-DW that only considers each flow’s packet arrival frequency. The results show that it outperforms the existing algorithms for the top-k frequent flow identification task. It provides nearly 98% identification accuracy with roughly 50% less memory cost. We also evaluate our PFD-DW algorithm for the more generalized task of identifying the recently persistent flows. It provides 93% identification rate of top-3000 persistent flows using only 576KB memory, while attaining 1.5% persistence estimation error. Its memory cost is reduced by 97% than another proposed solution PFD-SW under sliding time window. We have also developed a prototype of PFD-DW, based on the Fd.io VPP software switch accelerated by Intel DPDK.

Finding Recently Persistent Flows in High-Speed Packet Streams Based on Cuckoo Filter

Accurately Identify Time-decaying Heavy Hitters by Decay-aware Cuckoo Filter along Kicking Path

An Efficient Sampling And Classification Approach For Flow Detection In Sdn-Based Big Data Centers

FastKeeper: A Fast Algorithm for Identifying Top-k Real-time Large Flows

Persistent Sketch: A Memory-Efficient and Robust Algorithm for Finding Top-k Persistent Flows.

Heavykeeper: an Accurate Algorithm for Finding Top-K Elephant Flows

Tracking Millions of Flows in High Speed Networks for Application Identification

FloWatcher-DPDK: Lightweight Line-Rate Flow-Level Monitoring in Software

An Identification Method Combining Data Streaming Counting with Probabilistic Fading for Heavy-Hitter Flows

Dual Cuckoo Filter with a Low False Positive Rate for Deep Packet Inspection

Cuckoo Counter: Adaptive Structure of Counters for Accurate Frequency and Top-k Estimation

Identifying high-rate flows with less memory and time

A Generic Sketch for Estimating Super-Spreaders and Per-Flow Cardinality Distribution in High-Speed Data Streams

Probabilistic Fading Counting: an Efficient Identification Method for Finding Heavy Hitters

An adaptive method for identifying heavy hitters combining sampling and data streaming counting

Fast Detection of Heavy Hitters in Software Defined Networking Using an Adaptive and Learning Method

A novel spread estimation based abnormal flow detection in high-speed networks

Detecting Top-k Flows Combining Probabilistic Sketch and Sliding Window

FlowMon-DPDK: Parsimonious Per-Flow Software Monitoring at Line Rate

CuckooFlow - Achieving Fast Packet Classification for Virtual OpenFlow Switching by Exploiting Network Traffic Locality.