Abstract:A key function for network traffic monitoring and analysis is the ability to perform aggregate queries over multiple data streams. Change detection is an important primitive which can be extended to construct many aggregate queries. The recently proposed sketches [1] are among the very few that can detect heavy changes online for high speed links, and thus support various aggregate queries in both temporal and spatial domains. However, it does not preserve the keys (e.g., source IP address) of flows, making it difficult to reconstruct the desired set of anomalous keys. In an earlier abstract we proposed a framework for a reversible sketch data structure that offers hope for efficient extraction of keys [2]. However, this scheme is only able to detect a single heavy change key and places restrictions on the statistical properties of the key space. To address these challenges, we propose an efficient reverse hashing scheme to infer the keys of culprit flows from reversible sketches. There are two phases. The first operates online, recording the packet stream in a compact representation with negligible extra memory and few extra memory accesses. Our prototype single FPGA board implementation can achieve a throughput of over 16 Gbps for 40-byte-packet streams (the worst case). The second phase identifies heavy changes and their keys from the representation in nearly real time. We evaluate our scheme using traces from large edge routers with OC-12 or higher links. Both the analytical and experimental results show that we are able to achieve online traffic monitoring and accurate change/intrusion detection over massive data streams on high speed links, all in a manner that scales to large key space size. To the best of our knowledge, our system is the first to achieve these properties simultaneously.

A High-Performance Invertible Sketch for Network-Wide Superspreader Detection

SpreadSketch: Toward Invertible and Network-Wide Detection of Superspreaders

An Accurate and Invertible Sketch for Super Spread Detection

Enhancing Accuracy for Super Spreader Identification in High-Speed Data Streams

A Fast and Compact Invertible Sketch for Network-Wide Heavy Flow Detection

A hybrid local and distributed sketching design for accurate and scalable heavy key detection in network data streams

CardSketch: Shift Attention for Network-wide Cardinality Telemetry

LD-Sketch: A distributed sketching design for accurate and scalable anomaly detection in network data streams

Memory-Efficient and Flexible Detection of Heavy Hitters in High-Speed Networks.

Accurately Estimating User Cardinalities and Detecting Super Spreaders over Time

A novel spread estimation based abnormal flow detection in high-speed networks

SketchVisor: Robust Network Measurement for Software Packet Processing

Reversible Sketches: Enabling Monitoring and Analysis over High-Speed Data Streams

Detecting Stealthy Spreaders Using Online Outdegree Histograms

Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams

Monitoring Flow-level High-speed Data Streams with Reversible Sketches

Per-Flow Network Measurement With Distributed Sketch

Reverse Hashing for Sketch-based Change Detection on High-speed Networks

Dynamic Sketch: Efficient and Adjustable Heavy Hitter Detection for Software Packet Processing

Stalker Attacks: Imperceptibly Dropping Sketch Measurement Accuracy on Programmable Switches

DUNE: Improving Accuracy for Sketch-INT Network Measurement Systems