Abstract:Anomaly detection in distributed systems such as High-Performance Computing (HPC) clusters is vital for early fault detection, performance optimisation, security monitoring, reliability in general but also operational insights. Deep Neural Networks have seen successful use in detecting long-term anomalies in multidimensional data, originating for instance from industrial or medical systems, or weather prediction. A downside of such methods is that they require a static input size, or lose data through cropping, sampling, or other dimensionality reduction methods, making deployment on systems with variability on monitored data channels, such as computing clusters difficult. To address these problems, we present DeepHYDRA (Deep Hybrid DBSCAN/Reduction-Based Anomaly Detection) which combines DBSCAN and learning-based anomaly detection. DBSCAN clustering is used to find point anomalies in time-series data, mitigating the risk of missing outliers through loss of information when reducing input data to a fixed number of channels. A deep learning-based time-series anomaly detection method is then applied to the reduced data in order to identify long-term outliers. This hybrid approach reduces the chances of missing anomalies that might be made indistinguishable from normal data by the reduction process, and likewise enables the algorithm to be scalable and tolerate partial system failures while retaining its detection capabilities. Using a subset of the well-known SMD dataset family, a modified variant of the Eclipse dataset, as well as an in-house dataset with a large variability in active data channels, made publicly available with this work, we furthermore analyse computational intensity, memory footprint, and activation counts. DeepHYDRA is shown to reliably detect different types of anomalies in both large and complex datasets.

Time Series Anomaly Detection via Hypothesis Testing for Dynamical Systems

Anomaly Detection Algorithm Based on Electric Equipment

HybridAD: A Hybrid Model-Driven Anomaly Detection Approach for Multivariate Time Series

TimeSeriesBench: An Industrial-Grade Benchmark for Time Series Anomaly Detection Models

Robust Anomaly Detection for Time-series Data

DPDGAD: A Dual-Process Dynamic Graph-based Anomaly Detection for multivariate time series analysis in cyber-physical systems

An Enhancing Timeseries Anomaly Detection Using LSTM and Bi-LSTM Architectures

Multivariate Time Series Anomaly Detection via Dynamic Graph Forecasting

DeepHYDRA: Resource-Efficient Time-Series Anomaly Detection in Dynamically-Configured Systems

Automated Anomaly Detection Assisted by Discrimination Model for Time Series

Univariate Time Series Anomaly Detection Based on Hierarchical Attention Network

A Novel Deep Learning Approach for Anomaly Detection of Time Series Data

Developing an Unsupervised Real-Time Anomaly Detection Scheme for Time Series with Multi-Seasonality

Drift-aware Anomaly Detection for Non-stationary Time Series

SimAD: A Simple Dissimilarity-based Approach for Time Series Anomaly Detection

Algorithmic Recourse for Anomaly Detection in Multivariate Time Series

Multiple Hypothesis Testing for Anomaly Detection in Multi-type Event Sequences

An Empirical Analysis of Anomaly Detection Methods for Multivariate Time Series

Imputation-based Time-Series Anomaly Detection with Conditional Weight-Incremental Diffusion Models.

Self-adaptive Statistical Process Control for Anomaly Detection in Time Series.

Towards a General Time Series Anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoders