Abstract:In recent years, detecting anomalies in real-world computer networks has become a more and more challenging task due to the steady increase of high-volume, high-speed and high-dimensional streaming data, for which ground truth information is not available. Efficient detection schemes applied on networked embedded devices need to be fast and memory-constrained, and must be capable of dealing with concept drifts when they occur. Different approaches for unsupervised online outlier detection have been designed to deal with these circumstances in order to reliably detect malicious activity. In this paper, we introduce a novel framework called PCB-iForest, which generalized, is able to incorporate any ensemble-based online OD method to function on streaming data. Carefully engineered requirements are compared to the most popular state-of-the-art online methods with an in-depth focus on variants based on the widely accepted isolation forest algorithm, thereby highlighting the lack of a flexible and efficient solution which is satisfied by PCB-iForest. Therefore, we integrate two variants into PCB-iForest—an isolation forest improvement called extended isolation forest and a classic isolation forest variant equipped with the functionality to score features according to their contributions to a sample’s anomalousness. Extensive experiments were performed on 23 different multi-disciplinary and security-related real-world datasets in order to comprehensively evaluate the performance of our implementation compared with off-the-shelf methods. The discussion of results, including AUC, F1 score and averaged execution time metric, shows that PCB-iForest clearly outperformed the state-of-the-art competitors in 61% of cases and even achieved more promising results in terms of the tradeoff between classification and computational costs.

An efficient FPGA implementation of Mahalanobis distance-based outlier detection for streaming data

Continuous Angle-based Outlier Detection on High-dimensional Data Streams.

Real-time distance-based outlier detection in data streams

Energy-efficient Pipelined DTW Architecture on Hybrid Embedded Platforms

Continuous Outlier Mining of Streaming Data in Flink

Cube-based Incremental Outlier Detection for Streaming Computing

Attribute outlier detection over data streams

On the Improvement of the Isolation Forest Algorithm for Outlier Detection with Streaming Data

Design and Implementation of an FPGA-Based DNN Architecture for Real-time Outlier Detection

FCI-Outlier: An Efficient Frequent Closed Itemset-Based Outlier Detecting Approach on Data Stream

Robust High-dimensional Bioinformatics Data Streams Mining by ODR-ioVFDT

fSEAD: a Composable FPGA-based Streaming Ensemble Anomaly Detection Library

A Fast Outlier Detection Method for Big Data.

Outlier Detection in Streaming Data for Telecommunications and Industrial Applications: A Survey

An efficient approach for outlier detection from uncertain data streams based on maximal frequent patterns

FPGA Implementation of Feature Detection Algorithm Based on High Level Synthesis

Unsupervised real-time anomaly detection for streaming data

Fast Memory-efficient Anomaly Detection in Streaming Heterogeneous Graphs

Outlier detection by sampling with accuracy guarantees

Approximate Frequent Itemset Mining for Streaming Data on FPGA

Accelerating Frequent Item Counting with FPGA.