Abstract:Online monitoring user cardinalities (or degrees) in graph streams is fundamental for many applications. For example in a bipartite graph representing user-website visiting activities, user cardinalities (the number of distinct visited websites) are monitored to report network anomalies. These real-world graph streams may contain user-item duplicates and have a huge number of distinct user-item pairs, therefore, it is infeasible to exactly compute user cardinalities when memory and computation resources are <a class="link-external link-http" href="http://limited.Existing" rel="external noopener nofollow">this http URL</a> methods are designed to approximately estimate user cardinalities, whose accuracy highly depends on parameters that are not easy to set. Moreover, these methods cannot provide anytime-available estimation, as the user cardinalities are computed at the end of the data stream. Real-time applications such as anomaly detection require that user cardinalities are estimated on the fly. To address these problems, we develop novel bit and register sharing algorithms, which use a bit array and a register array to build a compact sketch of all users' connected items respectively. Compared with previous bit and register sharing methods, our algorithms exploit the dynamic properties of the bit and register arrays (e.g., the fraction of zero bits in the bit array at each time) to significantly improve the estimation accuracy, and have low time complexity (O(1)) to update the estimations each time they observe a new user-item pair. In addition, our algorithms are simple and easy to use, without requirements to tune any parameter. We evaluate the performance of our methods on real-world datasets. The experimental results demonstrate that our methods are several times more accurate and faster than state-of-the-art methods using the same amount of memory.

A Memory-Compact and Fast Sketch for Online Tracking Heavy Hitters in a Data Stream

Universal Online Sketch for Tracking Heavy Hitters and Estimating Moments of Data Streams

Highly Compact Virtual Counters For Per-Flow Traffic Measurement Through Register Sharing

Tree Sketch: an Accurate and Memory-Efficient Sketch for Network-Wide Measurement

In Search of a Memory-Efficient Framework for Online Cardinality Estimation

CountMax: A Lightweight and Cooperative Sketch Measurement for Software-Defined Networks

Diamond Sketch: Accurate Per-flow Measurement for Big Streaming Data

Short-Term Memory Sampling for Spread Measurement in High-Speed Networks

An Adaptive Counter-Splicing-Based Sketch for Efficient Per-Flow Size Measurement.

Flow Characteristics Estimation Based On Multilayer Virtual Active Counter Sharing in Data Center Network

Couper: Memory-Efficient Cardinality Estimation under Unbalanced Distribution.

TailoredSketch: A Fast and Adaptive Sketch for Efficient Per-Flow Size Measurement

Traffic Measurement For Big Network Data Introduction

Count-Less: A Counting Sketch for the Data Plane of High Speed Switches

One Memory Access Sketch: A More Accurate And Faster Sketch For Per-Flow Measurement

Memory-Efficient Performance Monitoring on Programmable Switches with Lean Algorithms

DISCO: Memory Efficient and Accurate Flow Statistics for Network Measurement

Hyper-Compact Virtual Estimators for Big Network Data Based on Register Sharing

Utilizing Dynamic Properties of Sharing Bits and Registers to Estimate User Cardinalities over Time

Self-Adaptive Sampling for Network Traffic Measurement

A novel spread estimation based abnormal flow detection in high-speed networks