Abstract:Internet has become an essential part of the daily life for billions of users worldwide, who are using a large variety of network services and applications everyday. However, there have been serious security problems and network failures that are hard to resolve, for example, Botnet attacks, polymorphic worm/virus spreading, DDoS, and flash crowds. To address many of these problems, we need to have a network-wide view of the traffic dynamics, and more importantly, be able to detect traffic anomaly in a timely manner. Existing network measurement and monitoring solutions often suffer scalability problems caused by overly large processing, space, or communication overhead. In this paper, we propose to develop sketch-based algorithms for network-wide anomaly detection that are able to detect both high-profile and coordinated low-profile traffic anomalies as an outlier in the regular traffic patterns. Our approach is based on the spatial analysis by using traffic measurements from multiple monitors. Spatial analysis methods have been proved to be effective in detecting network-wide traffic anomalies that are not detectable at a single monitor. To our knowledge, Principle Component Analysis (PCA) is the best-known spatial detection method for the coordinated low-profile traffic anomalies. However, existing PCA-based solutions have scalability problems in that they require O(m 2 n) running time and O(mn) space to analyze traffic measurements from m aggregated traffic flows within a sliding window of the length n, which makes it often infeasible to be deployed for monitoring large-scale high-speed networks. We propose two novel sketch-based algorithms for PCA-based traffic anomaly detection in a distributed fashion. Our algorithm can archive O(w log n) running time and O(w log 2 n) space at local monitors, and O(m 2 log n) running time and O(m log n) space at Network Operation Center, where w denotes the maximum number of traffic flows at a single local monitor. Additionally, our algorithm can protect the privacy of traffic measurements for Internet Service Providers.

A PCA-Based Traffic Monitoring Approach for Distributed Computing Systems

A Novel Change-Point Detection Approach for Monitoring High-Dimensional Traffics in Distributed Systems

A distributed data streaming algorithm for network-wide traffic anomaly detection

Sketch-Based Streaming PCA Algorithm for Network-Wide Traffic Anomaly Detection

PCA Based Approach for Monitoring the Spatial-and-temporal Characteristics of P2P Traffic

Sketch-based Network-wide Traffic Anomaly Detection

Distributed Parallel PCA for Modeling and Monitoring of Large-Scale Plant-Wide Processes with Big Data.

Communication-efficient online detection of network-wide anomalies

The abnormal traffic detection scheme based on PCA and SSH

An Effective Method Of Monitoring The Large-Scale Traffic Pattern Based On Rmt And Pca

A RMT and PCA-Based Method of Monitoring the Large-Scale Traffic Pattern

Traffic Data Analysis Using Kernel PCA and Self-Organizing Map

On-Line Anomaly Detection with High Accuracy.

Robust Traffic Anomaly Detection with Principal Component Pursuit.

Traffic Flow Decomposition and Prediction Based on Robust Principal Component Analysis

Structural Analysis of Network Traffic Matrix via Relaxed Principal Component Pursuit

Data-Driven Optimized Distributed Dynamic PCA for Efficient Monitoring of Large-Scale Dynamic Processes

Privacy-preserving anomaly detection across multi-domain networks

Statistical Analysis for Origin-Destination Traffic Anomalies

Structure Analysis of Network Traffic Matrix Based on Relaxed Principal Component Pursuit

A Probabilistic Principal Component Analysis Approach for Detecting Traffic Anomaly in Industrial Networks