Abstract:Due to the massive amount of data in high-speed network traffic and the limit on processing capability, it is a great challenge to accurately measure and monitor network traffic over high-speed links online. A new data structure is presented in this paper for locating the hosts associated with large connection degrees or significant changes in connection degrees based on the reversible connection degree sketch to monitor anomalous network traffic. The reversible connection degree sketch builds a compact summary of host connection degrees efficiently and accurately. For each packet coming, it only needs to set several bits selected in a bit array by a group of hash functions. These hash functions are designed based on the Chinese Remainder Theorem so that the in-degree or out-degree associated with a given host can be accurately estimated. With this new data structure, we develop a new reverse sketch method for locating abnormal hosts. Although the reversible connection degree sketch does not preserve any host address information, we can analytically reconstruct the host addresses associated with large connection degrees or significant changes in connection degrees by a simple calculation purely based on the characteristics of the hash functions. Furthermore, a reinforced reversible connection degree sketch, the double connection degree sketch, is developed to reduce false positives which are commonly encountered in the sketch-based methods. A traffic monitoring system based on this double connection degree is developed to detect and classify the abnormal hosts associated with large connection degrees or significant changes in connection degrees. The experiments are conducted based on the actual network traffic and the testing results show that our method is accurate and efficient.

A New Sketch Method for Measuring Host Connection Degree Distribution

A Data Streaming Method for Monitoring Host Connection Degrees of High-Speed Links

A New Virtual Indexing Method for Measuring Host Connection Degrees

Accelerating Sketch-based End-Host Traffic Measurement with Automatic DPU Offloading

A New Data Streaming Method for Locating Hosts with Large Connection Degree

Locality-Sensitive Sketching for Resilient Network Flow Monitoring

An effective and accurate flow size measurement using funnel-shaped sketch

A new connection degree calculation and measurement method for large scale network monitoring.

Unbiased Real-time Traffic Sketching

DAP-Sketch: An accurate and effective network measurement sketch with Deterministic Admission Policy

A Sketch Algorithm to Monitor High Packet Delay in Network Traffic

A survey of sketches in traffic measurement: Design, Optimization, Application and Implementation

Virtual indexing based methods for estimating node connection degrees

Enhanced Machine Learning Sketches for Network Measurements

Per-Flow Network Measurement With Distributed Sketch

A hybrid local and distributed sketching design for accurate and scalable heavy key detection in network data streams

SketchVisor: Robust Network Measurement for Software Packet Processing

An Accurate Estimation Algorithm for Big Data Streams.

Hierarchical Sketch: an Efficient Solution for Threshold-t Flows Measurement in High-Speed Networks

ALSketch: An adaptive learning-based sketch for accurate network measurement under dynamic traffic distribution

LD-Sketch: A distributed sketching design for accurate and scalable anomaly detection in network data streams