Abstract:Internet has become an essential part of the daily life for billions of users worldwide, who are using a large variety of network services and applications everyday. However, there have been serious security problems and network failures that are hard to resolve, for example, Botnet attacks, polymorphic worm/virus spreading, DDoS, and flash crowds. To address many of these problems, we need to have a network-wide view of the traffic dynamics, and more importantly, be able to detect traffic anomaly in a timely manner. Existing network measurement and monitoring solutions often suffer scalability problems caused by overly large processing, space, or communication overhead. In this paper, we propose to develop sketch-based algorithms for network-wide anomaly detection that are able to detect both high-profile and coordinated low-profile traffic anomalies as an outlier in the regular traffic patterns. Our approach is based on the spatial analysis by using traffic measurements from multiple monitors. Spatial analysis methods have been proved to be effective in detecting network-wide traffic anomalies that are not detectable at a single monitor. To our knowledge, Principle Component Analysis (PCA) is the best-known spatial detection method for the coordinated low-profile traffic anomalies. However, existing PCA-based solutions have scalability problems in that they require O(m 2 n) running time and O(mn) space to analyze traffic measurements from m aggregated traffic flows within a sliding window of the length n, which makes it often infeasible to be deployed for monitoring large-scale high-speed networks. We propose two novel sketch-based algorithms for PCA-based traffic anomaly detection in a distributed fashion. Our algorithm can archive O(w log n) running time and O(w log 2 n) space at local monitors, and O(m 2 log n) running time and O(m log n) space at Network Operation Center, where w denotes the maximum number of traffic flows at a single local monitor. Additionally, our algorithm can protect the privacy of traffic measurements for Internet Service Providers.

Sketch-based Network-wide Traffic Anomaly Detection

Sketch-Based Streaming PCA Algorithm for Network-Wide Traffic Anomaly Detection

A distributed data streaming algorithm for network-wide traffic anomaly detection

Detecting hidden anomalies using sketch for high-speed network data stream monitoring

On-Line Anomaly Detection with High Accuracy.

Communication-efficient online detection of network-wide anomalies

Privacy-preserving anomaly detection across multi-domain networks

On-Line Network Traffic Anomaly Detection Based on Tensor Sketch

Sketch-based change detection: methods, evaluation, and applications.

A PCA-Based Traffic Monitoring Approach for Distributed Computing Systems

A Novel Change-Point Detection Approach for Monitoring High-Dimensional Traffics in Distributed Systems

A Real-Time Detection Approach to Network Traffic Anomalies in Communication Networks

LD-Sketch: A distributed sketching design for accurate and scalable anomaly detection in network data streams

A flow-based anomaly detection method using sketch and combinations of traffic features

In-network PCA and anomaly detection

Detecting Distributed Network Traffic Anomaly with Network-Wide Correlation Analysis

An Adaptive Profile-Based Approach for Detecting Anomalous Traffic in Backbone.

Research on the detection of abnormal traffic for multi-channel network

Detecting traffic anomalies at application layer in metro area network

PCA-subspace Method - is It Good Enough for Network-Wide Anomaly Detection.

Multi‐scale Anomaly Detection for High‐speed Network Traffic