Abstract:Non-stationary distribution, in which the data distribution evolves over time, is a common issue in many application fields, e.g., intrusion detection and grid computing. Detecting the changes in massive streaming data with a non-stationary distribution helps to alarm the anomalies, to clean the noises, and to report the new patterns. In this paper, we employ a novel approach for detecting changes in streaming data with the purpose of improving the quality of modeling the data streams. Through observing the outliers, this approach of change detection uses a weighted standard deviation to monitor the evolution of the distribution of data streams. A cumulative statistical test, Page-Hinkley, is employed to collect the evidence of changes in distribution. The parameter used for reporting the changes is self-adaptively adjusted according to the distribution of data streams, rather than set by a fixed empirical value. The self-adaptability of the novel approach enhances the effectiveness of modeling data streams by timely catching the changes of distributions. We validated the approach on an online clustering framework with a benchmark KDDcup 1999 intrusion detection data set as well as with a real-world grid data set. The validation results demonstrate its better performance on achieving higher accuracy and lower percentage of outliers comparing to the other change detection approaches.

Survey on Change Detection over Data Stream

Detecting Change in Data Stream: Using Sampling Technique

An Algorithm for Data Stream Speed Anomaly Detection Based on RCSW

On Mining Time-Changing Data Streams

A Unifying Method For Outlier And Change Detection From Data Streams

Stream Event Detection: A Unified Framework for Mining Outlier, Change and Burst Simultaneously over Data Stream

Survey of Data Mining on Data Stream

Detecting Changes in Data Streams without Knowing the True Class Labels

A Novel Change Detecting Method for Monitoring Data Streams in Data Centers

Self-adaptive Change Detection in Streaming Data with Non-Stationary Distribution

Data stream analysis and research

Data mining : past present and future - a typical survey on data streams

Survey of the Study on Frequent Pattern Mining in Data Streams

Online Detection of Data Stream Changes Based on Maximum Frequent Itemset Entropy

A Survey of Mining Frequent Patterns in Data Streams

Research Overview of Related Techniques and Algorithms on Frequent Pattern Mining in Data Stream

A burst change detection algorithm for data streams

Research of Data Stream Mining Technology

A Novel Change-Point Detection Approach for Monitoring High-Dimensional Traffics in Distributed Systems

Concept Drift Detection in Data Stream Mining : A literature review

Survey of the Study on Frequent Pattern Mining in Data St Reams