Abstract:Engineers at eBay utilize robust methods in monitoring IT system signals for anomalies. However, the growing scale of signals, both in volumes and dimensions, overpowers traditional statistical state-space or supervised learning tools. Thus, state-of-the-art methods based on unsupervised deep learning are sought in recent research. However, we experienced flaws when implementing those methods, such as requiring partial supervision and weaknesses to high dimensional datasets, among other reasons discussed in this paper. We propose a practical approach for inferring anomalies from large multivariate sets. We observe an abundance of time series in real-world applications, which exhibit asynchronous and consistent repetitive variations, such as IT, weather, utility, and transportation. Our solution is designed to leverage this behavior. The solution utilizes spectral analysis on the latent representation of a pre-trained autoencoder to extract dominant frequencies across the signals, which are then used in a subsequent network that learns the phase shifts across the signals and produces a synchronized representation of the raw multivariate. Random subsets of the synchronous multivariate are then fed into an array of autoencoders learning to minimize the quantile reconstruction losses, which are then used to infer and localize anomalies based on a majority vote. We benchmark this method against state-of-the-art approaches on public datasets and eBay's data using their referenced evaluation methods. Furthermore, we address the limitations of the referenced evaluation methods and propose a more realistic evaluation method.

Low Latency Anomaly Detection and Bayesian Network Prediction of Anomaly Likelihood

An Enhancing Timeseries Anomaly Detection Using LSTM and Bi-LSTM Architectures

Low-count Time Series Anomaly Detection

Anomaly Prediction: A Novel Approach with Explicit Delay and Horizon

A Survey on Anomaly Detection for Technical Systems using LSTM Networks

HybridAD: A Hybrid Model-Driven Anomaly Detection Approach for Multivariate Time Series

Monte Carlo EM for Deep Time Series Anomaly Detection

LSTM-based Anomaly Detection for Non-linear Dynamical System

Practical Approach to Asynchronous Multivariate Time Series Anomaly Detection and Localization

Anomaly Detection for Time Series Using VAE-LSTM Hybrid Model

Bayesian Optimization with Machine Learning Algorithms Towards Anomaly Detection

Anomaly Scoring for Prediction-Based Anomaly Detection in Time Series

A hybrid IndRNNLSTM approach for real-time anomaly detection in software-defined networks

Community detection and anomaly prediction in dynamic networks

Latent Anomaly Detection Through Density Matrices

Probabilistic Models for Anomaly Detection in Remote Sensor Data Streams

Machine Learning for Real-Time Anomaly Detection in Optical Networks

An efficient algorithm for anomaly detection in a flight system using dynamic bayesian networks

Intrinsic Anomaly Detection for Multi-Variate Time Series

Detection of Anomalies in Multivariate Time Series Using Ensemble Techniques