Abstract:The scale and complexity of today's networks are increasing at a staggering pace, and so are the characteristics of data traffic and diverse applications or services in the networks. Their interdependencies also become more and more complicated, which ask for advanced network traffic measurement and analysis techniques. Besides packet level and flow level statistics, it is also important to monitor and understand the behavior of network users and applications, from the perspective of how they communicate with each other. For example, a popular server may attract a lot of connections from interested users; P2P peers often form clusters with intensive communications with each other; Botnet zombies receive regular commands from their botmaster and they may join a malicious campaign later to spread out a mass of spam emails or launch a DDoS attack. Such high level communication patterns as massive concurrent connections or causality of events are often useful behavior signatures of certain applications, or act as indications of anomaly. They can be very helpful in network management, traffic engineering, application behavior analysis, and anomaly detection. In this thesis, we study three interesting and useful communication patterns, including top spreaders, top scanners, and flow correlations. They have practical usage, especially in network management and anomaly detection. However, there is very little support from the network itself for high quality measurement of such non-trivial statistics, and the ever-increasing link speed and traffic volume have brought even greater challenges to our measurement and analysis. We take the approach of data streaming algorithms. First, we propose a general scheme called multiplexed sketches to efficiently estimate statistics of a large number of streams. Then we design appropriate algorithms that can accompany the multiplexed sketches to efficiently track each of the three communication patterns we have proposed. Particularly, we design a general "filter-tracker-digester" framework, where the filter provides a rough statistics estimation, the tracker tracks the IDs of potential candidate spreaders or scanners, and the digester is implemented as multiplexed sketches for accurate statistics estimation. Several challenges are addressed in our design, including traffic scale, skewness, speed, memory usage, and result accuracy. The performance of our algorithms is analyzed both mathematically and experimentally. We show they can achieve accuracy and speed of at least an order of magnitude higher than alternative approaches.

A novel spread estimation based abnormal flow detection in high-speed networks

Spread Estimation with Non-Duplicate Sampling in High-Speed Networks

An Efficient K-Persistent Spread Estimator for Traffic Measurement in High-Speed Networks

You Can Drop but You Can't Hide: <tex>$K$</tex>-persistent Spread Estimation in High-speed Networks

Multi-layer Adaptive Sampling for Per-Flow Spread Measurement

Estimating the Persistent Spreads in High-Speed Networks

Randomized Error Removal for Online Spread Estimation in High-Speed Networks

A Generic Sketch for Estimating Super-Spreaders and Per-Flow Cardinality Distribution in High-Speed Data Streams

MIME: Fast and Accurate Flow Information Compression for Multi-Spread Estimation

Enhancing Accuracy for Super Spreader Identification in High-Speed Data Streams

Online High-Cardinality Flow Detection over Big Network Data Stream.

Short-Term Memory Sampling for Spread Measurement in High-Speed Networks

Towards Accurate and Efficient Super Spreader Detection with Sketching

KTSketch: Finding K-Persistent T-Spread Flows in High-Speed Networks

Real-time Spread Burst Detection in Data Streaming

An Online Framework for Catching Top Spreaders and Scanners.

Randomized Error Removal for Online Spread Estimation in Data Streaming.

Efficient tracking of significant communication patterns in computer networks

Detecting hidden anomalies using sketch for high-speed network data stream monitoring

Self-Adaptive Sampling for Network Traffic Measurement

Hierarchical Sketch: an Efficient Solution for Threshold-t Flows Measurement in High-Speed Networks