Abstract:Correlated anomaly detection (CAD) from streaming data is a type of group anomaly detection and an essential task in useful real-time data mining applications like botnet detection, financial event detection, industrial process monitor, etc. The primary approach for this type of detection in previous researches is based on principal score (PS) of divided batches or sliding windows by computing top eigenvalues of the correlation matrix, e.g. the Lanczos algorithm. However, this paper brings up the phenomenon of principal score degeneration for large data set, and then mathematically and practically prove current PS-based methods are likely to fail for CAD on large-scale streaming data even if the number of correlated anomalies grows with the data size at a reasonable rate; in reality, anomalies tend to be the minority of the data, and this issue can be more serious. We propose a framework with two novel randomized algorithms rPS and gPS for better detection of correlated anomalies from large streaming data of various correlation strength. The experiment shows high and balanced recall and estimated accuracy of our framework for anomaly detection from a large server log data set and a U.S. stock daily price data set in comparison to direct principal score evaluation and some other recent group anomaly detection algorithms. Moreover, our techniques significantly improve the computation efficiency and scalability for principal score calculation.

Application of Anomaly Detection in Alert Correlation Analysis

Detecting Distributed Network Traffic Anomaly with Network-Wide Correlation Analysis

Correlation Analysis for Distributed Intrusion Alert

Discovering Anomaly on the Basis of Flow Estimation of Alert Feature Distribution

MTS Anomaly Detection Based on Temporal and Feature Correlation

Improved correlation analysis and visualization of industrial alarm data

The Network-Wide Multi-Traffic Correlative Anomaly Detection

Non-regular alarm correlation analysis

Network Anomaly Detection Based on Statistical Approach and Time Series Analysis

Alert Correlation by A Retrospective Method

Survey of the Security Alerts Correlation Algorithms

Anomaly detection of industrial state quantity time-Series data based on correlation and long short-term memory

Spatio-Temporal Correlation Analysis of Online Monitoring Data for Anomaly Detection and Location in Distribution Networks

An Anomaly Detection Approach for Multiple Monitoring Data Series Based on Latent Correlation Probabilistic Model.

MAAC: Novel Alert Correlation Method To Detect Multi-step Attack

Correlation-Based Method for Tracing Multi-dimensional Time Series Data Anomalies

Correlated Anomaly Detection from Large Streaming Data

The Research of Alert Correlation Based on Attack Intetion

An Algorithm of Alert Correlation Based on Address Correlation Graph in Distributed Intrusion Detection System

An Anomaly Detection and Analysis Method for Network Traffic Based on Correlation Coefficient Matrix.

Alert Correlation Based on Description Logics