Abstract:The identification of anomalous activities is a challenging and crucially important task in sensor networks. This task is becoming increasingly complex with the increasing volume of data generated in real-world domains, and greatly benefits from the use of predictive models to identify anomalies in real time. A key use case for this task is the identification of misbehavior that may be caused by involuntary faults or deliberate actions. However, currently adopted anomaly detection methods are often affected by limitations such as the inability to analyze large-scale data, a reduced effectiveness when data presents multiple densities, a strong dependence on user-defined threshold configurations, and a lack of explainability in the extracted predictions. In this paper, we propose a distributed deep learning method that extends growing hierarchical self-organizing maps, originally designed for clustering tasks, to address anomaly detection tasks. The SOM-based modeling capabilities of the method enable the analysis of data with multiple densities, by exploiting multiple SOMs organized as a hierarchy. Our map-reduce implementation under Apache Spark allows the method to process and analyze large-scale sensor network data. An automatic threshold-tuning strategy reduces user efforts and increases the robustness of the method with respect to noisy instances. Moreover, an explainability component resorting to instance-based feature ranking emphasizes the most salient features influencing the decisions of the anomaly detection model, supporting users in their understanding of raised alerts. Experiments are conducted on five real-world sensor network datasets, including wind and photovoltaic energy production, vehicular traffic, and pedestrian flows. Our results show that the proposed method outperforms state-of-the-art anomaly detection competitors. Furthermore, a scalability analysis reveals that the method is able to scale linearly as the data volume presented increases, leveraging multiple worker nodes in a distributed computing setting. Qualitative analyses on the level of anomalous pollen in the air further emphasize the effectiveness of our proposed method, and its potential in determining the level of danger in raised alerts.

Developing Big Data Anomaly Dynamic and Static Detection Algorithms: AnomalyDSD Spark Package

Development of infrastructure for anomalies detectionin big data

Regression-based Online Anomaly Detection for Smart Grid Data

Anomaly Detection in Big Data

Distributed and explainable GHSOM for anomaly detection in sensor networks

KMDT: A Hybrid Cluster Approach for Anomaly Detection Using Big Data

Anomaly detection of unstructured big data via semantic analysis and dynamic knowledge graph construction

Entropy-based Outlier Detection Using Spark

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

Research on Dynamic Data Flow Anomaly Detection based on Machine Learning

Anomaly Detection at Scale: The Case for Deep Distributional Time Series Models

Anomaly Detection for Industrial Big Data

DeepHYDRA: Resource-Efficient Time-Series Anomaly Detection in Dynamically-Configured Systems

DPDGAD: A Dual-Process Dynamic Graph-based Anomaly Detection for multivariate time series analysis in cyber-physical systems

A DISTRIBUTED AND PARALLEL ANOMALY DETECTION IN HYPERSPECTRAL IMAGES BASED ON LOW-RANK AND SPARSE REPRESENTATION

Dynamic Micro-cluster-Based Streaming Data Clustering Method for Anomaly Detection.

SDOoop: Capturing Periodical Patterns and Out-of-phase Anomalies in Streaming Data Analysis

A Survey of Graph-based Deep Learning for Anomaly Detection in Distributed Systems

Guest Editorial Introduction to the Special Issue on Anomaly Detection in Emerging Data-Driven Applications: Theory, Algorithms, and Applications

anomaly: Detection of Anomalous Structure in Time Series Data

Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Values