Abstract:Maintaining the high availability of online systems requires reliable and fast online anomaly detection and fault localization. However, existing anomaly detection methods either suffer high training costs and low generalization capabilities or are designed and evaluated using offline data with limited efficacy in online usage. Furthermore, these methods' fault localization capabilities are often inadequate due to external observability constraints. Therefore, designing a new approach to address these limitations effectively is essential. To address the aforementioned limitations, this paper proposes a novel non-intrusive in situ anomaly detection and fault lo-calization framework, Detective-Dee. The proposed framework leverages a compressed sensing method for anomaly detection, which exhibits strong generalization capabilities and eliminates extensive training. Detective-Dee further improves its performance by incorporating three optimization techniques: concurrent sub-stitution sampling, Look-Up-Table-based similarity calculation, and substitution window-based threshold selection to improve parallelism and reduce computational and comparison overheads. Additionally, the framework adopts an innovative non-intrusive fault localization strategy based on anomaly detection triggering. This approach utilizes the dynamic instrumentation capabilities of eBPF, combined with extracting vulnerable function and function call chains through source code analysis, to improve the online anomaly detection capability and achieve robust fault localization with low overhead. To validate the effectiveness of Detective-Dee, we developed a prototype system and conducted a comprehensive evaluation. The results demonstrate that, compared to the state-of-the-art anomaly detection method, Detective-Dee exhibits a 4x improve-ment in anomaly detection speed while maintaining higher online and comparable offline detection ability. Furthermore, under 33 real-world fault cases across eight popular distributed systems, Detective-Dee successfully detects 31 cases and accurately locates 26 cases with less than 1% overhead, outperforming the state-of-the-art method.

Anomaly Detection and Reporting System for POS

An Unsupervised Framework for Anomaly Detection in a Water Treatment System

An Anomaly Detection Algorithm for the Power Industrial Terminal Security Monitoring

Detective-Dee: A Non-Intrusive in Situ Anomaly Detection and Fault Localization Framework

Deep-Compact-Clustering Based Anomaly Detection Applied to Electromechanical Industrial Systems

Anomaly detection system for network transport with machine learning approach

Anomaly Detection based on 1D-CNN-LSTM Auto-Encoder for Bearing Data

An Unsupervised Anomaly Detection Approach Based On Industrial Big Data

A Stacked Autoencoder Neural Network based Automated Feature Extraction Method for Anomaly detection in On-line Condition Monitoring

A Smart-Anomaly-Detection System for Industrial Machines Based on Feature Autoencoder and Deep Learning

Real-Time Predictive Maintenance using Autoencoder Reconstruction and Anomaly Detection

Dirichlet Probability Navigated Fault Detection Via Key-Group Memory Auto-Encoder Under Non-Stationary Working Conditions

ML-based Anomaly Detection in Optical Fiber Monitoring

Anomaly Detection and Interpretation using Multimodal Autoencoder and Sparse Optimization

Anomaly Detection of Smart Metering System for Power Management with Battery Storage System/Electric Vehicle

An anomaly prediction framework for financial IT systems using hybrid machine learning methods

Semi-supervised Anomaly Detection using AutoEncoders

Anomaly Detection for Sensor Signals Utilizing Deep Learning Autoencoder-Based Neural Networks

Multimodal Anomaly Detection based on Deep Auto-Encoder for Object Slip Perception of Mobile Manipulation Robots

System-level Hardware Failure Prediction Using Deep Learning

Practical autoencoder based anomaly detection by using vector reconstruction error