Abstract:Detection of anomalous situations for complex mission-critical systems holds paramount importance when their service continuity needs to be ensured. A major challenge in detecting anomalies from the operational data arises due to the imbalanced class distribution problem since the anomalies are supposed to be rare events. This paper evaluates a diverse array of machine learning-based anomaly detection algorithms through a comprehensive benchmark study. The paper contributes significantly by conducting an unbiased comparison of various anomaly detection algorithms, spanning classical machine learning including various tree-based approaches to deep learning and outlier detection methods. The inclusion of 104 publicly available and a few proprietary industrial systems datasets enhances the diversity of the study, allowing for a more realistic evaluation of algorithm performance and emphasizing the importance of adaptability to real-world scenarios. The paper dispels the deep learning myth, demonstrating that though powerful, deep learning is not a universal solution in this case. We observed that recently proposed tree-based evolutionary algorithms outperform in many scenarios. We noticed that tree-based approaches catch a singleton anomaly in a dataset where deep learning methods fail. On the other hand, classical SVM performs the best on datasets with more than 10% anomalies, implying that such scenarios can be best modeled as a classification problem rather than anomaly detection. To our knowledge, such a study on a large number of state-of-the-art algorithms using diverse data sets, with the objective of guiding researchers and practitioners in making informed algorithmic choices, has not been attempted earlier.

Tree-based algorithms for weakly supervised anomaly detection

Back To The Roots: Tree-Based Algorithms for Weakly Supervised Anomaly Detection

Anomaly Detection in Presence of Irrelevant Features

Incorporating Physical Priors into Weakly-Supervised Anomaly Detection

Universal Anomaly Detection at the LHC: Transforming Optimal Classifiers and the DDD Method

Searches for the BSM scenarios at the LHC using decision tree based machine learning algorithms: A comparative study and review of Random Forest, Adaboost, XGboost and LightGBM frameworks

The interplay of machine learning-based resonant anomaly detection methods

Complete Optimal Non-Resonant Anomaly Detection

Searches for the BSM scenarios at the LHC using decision tree-based machine learning algorithms: a comparative study and review of random forest, AdaBoost, XGBoost and LightGBM frameworks

Unravelling physics beyond the standard model with classical and quantum anomaly detection

The Interplay of Machine Learning--based Resonant Anomaly Detection Methods

Anomaly detection in collider physics via factorized observables

Can Tree Based Approaches Surpass Deep Learning in Anomaly Detection? A Benchmarking Study

Combining resonant and tail-based anomaly detection

Robust Anomaly Detection for Particle Physics Using Multi-Background Representation Learning

The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider

Non-resonant anomaly detection with background extrapolation

Machine Learning for Anomaly Detection in Particle Physics

Exploring Optimal Transport for Event-Level Anomaly Detection at the Large Hadron Collider

GAN-AE : An anomaly detection algorithm for New Physics search in LHC data