Abstract:The widespread use of Internet-of-Things (IoT) technologies, smartphones, and social media services generates huge amounts of data streaming at high velocity. Automatic interpretation of these rapidly arriving data streams is required for the timely detection of interesting events that usually emerge in the form of clusters. This article proposes a new relative of the visual assessment of the cluster tendency (VAT) model, which produces a record of structural evolution in the data stream by building a cluster heat map of the entire processing history in the stream. The existing VAT-based algorithms for streaming data, called inc-VAT/inc-iVAT and dec-VAT/dec-iVAT, are not suitable for high-velocity and high-volume streaming data because of high memory requirements and slower processing speed as the accumulated data increases. The scalable iVAT (siVAT) algorithm can handle big batch data, but for streaming data, it needs to be (re)applied everytime a new datapoint arrives, which is not feasible due to the associated computation complexities. To address this problem, we propose an incremental siVAT algorithm, called inc-siVAT, which deals with the streaming data in chunks. It first extracts a small size smart sample using an intelligent sampling scheme, called maximin random sampling (MMRS), then incrementally updates the smart sample points on the fly, using our novel incremental MMRS (inc-MMRS) algorithm, to reflect changes in the data stream after each chunk is processed, and finally, produces an incrementally built iVAT image of the updated smart sample, using the inc-VAT/inc-iVAT and dec-VAT/dec-iVAT algorithms. These images can be used to visualize the evolving cluster structure and for anomaly detection in streaming data. Our method is illustrated with one synthetic and four real datasets, two of which evolve significantly over time. Our numerical experiments demonstrate the algorithm's ability to successfully identify anomalies and visualize-changing cluster structure in streaming data.

Anomaly Detection over Streaming Data : Indy 500

Anomaly Detection over Streaming Data: Indy500 Case Study

Deep Learning for Anomaly Detection in IoT Systems: Techniques, Applications, and Future Directions

Visual Structural Assessment and Anomaly Detection for High-Velocity Data Streams

Anomaly Detection Models for IoT Time Series Data

Anomaly Detection for Industrial Big Data

Anomaly Detection of Streaming Data Based on Deep Learning

Unsupervised real-time anomaly detection for streaming data

Efficient Real-Time Decision Making Using Streaming Data Analytics in IoT Environment

Continuous Angle-based Outlier Detection on High-dimensional Data Streams.

Fast Memory-efficient Anomaly Detection in Streaming Heterogeneous Graphs

Real-Time Anomaly Detection for Streaming Analytics

Edge computing empowered anomaly detection framework with dynamic insertion and deletion schemes on data streams

A Human-in-the-Loop Anomaly Detection Architecture for Big Traffic Data of Cellular Network

Anomaly Detection in Streaming Sensor Data

ADDAI: Anomaly Detection using Distributed AI

Online Internet Traffic Monitoring and DDoS Attack Detection Using Big Data Frameworks.

Outlier Detection in Streaming Data for Telecommunications and Industrial Applications: A Survey

Anomaly Detection in Residential Video Surveillance on Edge Devices in IoT Framework

Dynamic Micro-cluster-Based Streaming Data Clustering Method for Anomaly Detection.

IoT streaming data integration from multiple sources