Abstract:In this paper, we consider the problem of thresholded monitoring in distributed data streams, that is, given multiple distributed data streams observed by multiple monitors during a certain period, finding the items whose global frequencies overall data streams exceeding a given threshold. We first derive a lower bound of communication overhead for any deterministic algorithm for this problem. Then, we propose two different schemas, i.e., Low-threshold Cascaded Cuckoo Filter (L-CCF) for low-threshold monitoring and High-threshold Cascaded Cuckoo Filter (H-CCF) for high-threshold monitoring. L-CCF and H-CCF can identify items whose frequency are more than the given threshold while a desired false negative rate (FNR) is achieved and communication overhead is optimized. The key idea is to compress the communication overhead caused by transferring the ID and frequency information at the same time. First, to reduce the communication overhead of transferring IDs, we propose to encode the IDs into separate tiny parts and store these tiny parts in L-CCF or H-CCF. Second, to reduce the communication overhead of transferring frequencies, we adopt carry-in counter technique in L-CCF and multiple sampling technique in H-CCF. We evaluated L-CCF and H-CCF on two real-world traces and compared their performance with two prior adapted algorithms. Our experimental results show that on average, L-CCF and H-CCF achieve FNRs with 55.7% and 65.56% better than that of comparison algorithms while FPRs is maintained at the level of 2.23%.

An adaptive method for identifying heavy hitters combining sampling and data streaming counting

An Identification Method Combining Data Streaming Counting with Probabilistic Fading for Heavy-Hitter Flows

Probabilistic Fading Counting: an Efficient Identification Method for Finding Heavy Hitters

An Efficient Sampling And Classification Approach For Flow Detection In Sdn-Based Big Data Centers

Accurate and Efficient Traffic Monitoring Using Adaptive Non-Linear Sampling Method

Learning-Based Heavy Hitters and Flow Frequency Estimation in Streams

Heavykeeper: an Accurate Algorithm for Finding Top-K Elephant Flows

Enhancing Accuracy for Super Spreader Identification in High-Speed Data Streams

Fast Detection of Heavy Hitters in Software Defined Networking Using an Adaptive and Learning Method

Fine-grained Probability Counting for Cardinality Estimation of Data Streams.

Time Stratified Adaptive Packet Sampling Based on Traffic Load

A novel spread estimation based abnormal flow detection in high-speed networks

An Online Framework for Catching Top Spreaders and Scanners.

ANLS: Adaptive Non-Linear Sampling Method for Accurate Flow Size Measurement.

An Adaptive Strategy for High-Speed Network Flow Compression

Poster : Compressing Flow Statistic Counters

Adaptive algorithm of identifying and measuring elephant flow based on least recently used replacement

Survey of Identifying Method for Measurement of Heavy-Hitter Traffic Flow

Thresholded Monitoring in Distributed Data Streams

The hardware implementation of adaptive non-linear sampling algorithm