Abstract:Anomaly detection to recognize unusual events in large scale systems in a time sensitive manner is critical in many industries, eg. bank fraud, enterprise systems, medical alerts, etc. Large-scale systems often grow in size and complexity over time, and anomaly detection algorithms need to adapt to changing structures. A hierarchical approach takes advantage of the implicit relationships in complex systems and localized context. The features in complex systems may vary drastically in data distribution, capturing different aspects from multiple data sources, and when put together provide a more complete view of the system. In this paper, two datasets are considered, the 1st comprising of system metrics from machines running on a cloud service, and the 2nd of application metrics from a large-scale distributed software system with inherent hierarchies and interconnections amongst its system nodes. Comparing algorithms, across the changepoint based PELT algorithm, cognitive learning-based Hierarchical Temporal Memory algorithms, Support Vector Machines and Conditional Random Fields provides a basis for proposing a Hierarchical Global-Local Conditional Random Field approach to accurately capture anomalies in complex systems across various features. Hierarchical algorithms can learn both the intricacies of specific features, and utilize these in a global abstracted representation to detect anomalous patterns robustly across multi-source feature data and distributed systems. A graphical network analysis on complex systems can further fine-tune datasets to mine relationships based on available features, which can benefit hierarchical models. Furthermore, hierarchical solutions can adapt well to changes at a localized level, learning on new data and changing environments when parts of a system are over-hauled, and translate these learnings to a global view of the system over time.

Hierarchical Temporal Memory method for time-series-based anomaly detection

Grid HTM: Hierarchical Temporal Memory for Anomaly Detection in Videos

An Anomaly Detection Approach Based on Bidirectional Temporal Convolutional Network and Multi-Head Attention Mechanism

Hierarchical Context Representation and Self-adaptive Thresholding for Multivariate Anomaly Detection

Temporal Convolutional Networks for Anomaly Detection in Time Series

Univariate Time Series Anomaly Detection Based on Hierarchical Attention Network

Multiple Temporal Context Embedding Networks for Unsupervised Time Series Anomaly Detection

Anomaly Detection of Spacecraft Telemetry Data Using Temporal Convolution Network

A Hierarchical Approach to Conditional Random Fields for System Anomaly Detection

Adaptive Memory Broad Learning System for Unsupervised Time Series Anomaly Detection

Deep Generative model with Hierarchical Latent Factors for Time Series Anomaly Detection

A Novel Deep Learning Approach for Anomaly Detection of Time Series Data

Disease Diagnosis Supported By Hierarchical Temporal Memory

Anomaly Detection in Heterogeneous Time Series Data for Server-Monitoring Tasks.

Online Intrusion Scenario Discovery and Prediction Based on Hierarchical Temporal Memory (HTM)

Time Series Anomaly Detection Based on CEEMDAN and LSTM.

MTS Anomaly Detection Based on Temporal and Feature Correlation

A novel unsupervised framework for time series data anomaly detection via spectrum decomposition

Hybrid Graph Transformer Networks for Multivariate Time Series Anomaly Detection

Sequential Anomaly Detection Based on Temporal-Difference Learning: Principles, Models and Case Studies

An Unsupervised Anomaly Detection Algorithm for Time Series Big Data