Abstract:The nature of modern data is increasingly real-time, making outlier detection crucial in any data-related field, such as finance for fraud detection and healthcare for monitoring patient vitals. Traditional outlier detection methods, such as the Local Outlier Factor (LOF) algorithm, struggle with real-time data due to the need for extensive recalculations with each new data point, limiting their application in real-time environments. While the Incremental LOF (ILOF) algorithm has been developed to tackle the challenges of online anomaly detection, it remains computationally expensive when processing large streams of data points, and its detection performance may degrade after a certain threshold of points have streamed in. In this paper, we propose a novel approach to enhance the efficiency of LOF algorithms for online anomaly detection, named the Efficient Incremental LOF (EILOF) algorithm. The EILOF algorithm only computes the LOF scores of new points without altering the LOF scores of existing data points. Although exact LOF scores have not yet been computed for the existing points in the new algorithm, datasets often contain noise, and minor deviations in LOF score calculations do not necessarily degrade detection performance. In fact, such deviations can sometimes enhance outlier detection. We systematically tested this approach on both simulated and real-world datasets, demonstrating that EILOF outperforms ILOF as the volume of streaming data increases across various scenarios. The EILOF algorithm not only significantly reduces computational costs, but also systematically improves detection accuracy when the number of additional points increases compared to the ILOF algorithm.

Continuous Kernel-Based Outlier Detection Over Distributed Data Streams

Continuous Adaptive Outlier Detection on Distributed Data Streams.

Continuous Angle-based Outlier Detection on High-dimensional Data Streams.

Efficient Outlier Detection Algorithm for Heterogeneous Data Streams.

Scalable distance-based outlier detection over high-volume data streams

Scalable KDE-based Top-N Local Outlier Detection over Large-Scale Data Streams

Outlier Detection Algorithms and their Availability to Data Streams Mining

Continuous Outlier Monitoring on Uncertain Data Streams

Frod: Fast And Robust Distance-Based Outlier Detection With Active-Inliers-Patterns In Data Streams

Online Reactive Anomaly Detection over Stream Data

An efficient algorithm for distributed density-based outlier detection on big data.

Research on Outlier Detection for High Dimensional Data Stream

Continuous Outlier Mining of Streaming Data in Flink

Top-k Outlier Detection Algorithm on Uncertain Data Stream

A Unifying Method For Outlier And Change Detection From Data Streams

Outlier detection in graph streams

An Efficient Outlier Detection Algorithm for Data Streaming

Anomaly Detection over Concept Drifting Data Streams

A Review of Local Outlier Factor Algorithms for Outlier Detection in Big Data Streams

Fast Mining Outliers in Online Data Streams

A Fast Outlier Detection Algorithm for Data Streams Based on Dynamic Grids