Abstract:Several techniques for multivariate time series anomaly detection have been proposed recently, but a systematic comparison on a common set of datasets and metrics is lacking. This paper presents a systematic and comprehensive evaluation of unsupervised and semi-supervised deep-learning based methods for anomaly detection and diagnosis on multivariate time series data from cyberphysical systems. Unlike previous works, we vary the model and post-processing of model errors, i.e. the scoring functions independently of each other, through a grid of 10 models and 4 scoring functions, comparing these variants to state of the art methods. In time-series anomaly detection, detecting anomalous events is more important than detecting individual anomalous time-points. Through experiments, we find that the existing evaluation metrics either do not take events into account, or cannot distinguish between a good detector and trivial detectors, such as a random or an all-positive detector. We propose a new metric to overcome these drawbacks, namely, the composite F-score ($Fc_1$), for evaluating time-series anomaly detection. Our study highlights that dynamic scoring functions work much better than static ones for multivariate time series anomaly detection, and the choice of scoring functions often matters more than the choice of the underlying model. We also find that a simple, channel-wise model - the Univariate Fully-Connected Auto-Encoder, with the dynamic Gaussian scoring function emerges as a winning candidate for both anomaly detection and diagnosis, beating state of the art algorithms.

Anomaly Detection in Univariate Time-series: A Survey on the State-of-the-Art

Anomaly Detection Algorithm Based on Electric Equipment

Do Deep Neural Networks Contribute to Multivariate Time Series Anomaly Detection?

A Comparative Study on Unsupervised Anomaly Detection for Time Series: Experiments and Analysis

Unsupervised Anomaly Detection in Time-series: An Extensive Evaluation and Analysis of State-of-the-art Methods

An Experimental Evaluation of Anomaly Detection in Time Series

Deep Learning for Time Series Anomaly Detection: A Survey

A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data

Multivariate Time Series Anomaly Detection: Fancy Algorithms and Flawed Evaluation Methodology

An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series

Online Model-based Anomaly Detection in Multivariate Time Series: Taxonomy, Survey, Research Challenges and Future Directions

An Empirical Analysis of Anomaly Detection Methods for Multivariate Time Series

Dive into Time-Series Anomaly Detection: A Decade Review

An Unsupervised Anomaly Detection Algorithm for Time Series Big Data

Experimental Comparison and Survey of Twelve Time Series Anomaly Detection Algorithms

Is it worth it? Comparing six deep and classical methods for unsupervised anomaly detection in time series

CANet: Convolutional Attention-based Network for Multivariate Time Series Anomaly Detection

Unsupervised Anomaly Detection for IoT-Based Multivariate Time Series: Existing Solutions, Performance Analysis and Future Directions

Multivariate Time Series Anomaly Detection with Few Positive Samples

Deep Learning for Anomaly Detection: A Survey