A Comprehensive Study of Machine Learning Techniques for Log-Based Anomaly Detection

Shan Ali,Chaima Boufaied,Domenico Bianculli,Paula Branco,Lionel Briand

2024-05-20

Abstract:Growth in system complexity increases the need for automated techniques dedicated to different log analysis tasks such as Log-based Anomaly Detection (LAD). The latter has been widely addressed in the literature, mostly by means of a variety of deep learning techniques.

Software Engineering,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate and compare the performance of different types of machine learning (ML) techniques in log - based anomaly detection (LAD). Specifically, the author believes that existing research mainly focuses on deep - learning techniques while ignoring the potential of traditional machine - learning techniques and semi - supervised learning techniques. In addition, existing evaluation methods usually only focus on detection accuracy while ignoring other important performance indicators, such as time performance and sensitivity to hyper - parameter tuning.

Therefore, the goal of this paper is to comprehensively evaluate and compare a variety of supervised and unsupervised traditional and deep - learning techniques through systematic empirical research, considering the following four evaluation criteria:

1. **Detection accuracy**: Measures the ability of the model to detect anomalies.
2. **Time performance**: Includes training time and prediction time.
3. **Sensitivity of detection accuracy and time performance to hyper - parameter tuning**: Evaluates the stability and performance of the model under different hyper - parameter settings.

### Research motivation
1. **Expand the range of techniques**: Most previous studies have focused on deep - learning techniques, but traditional machine - learning techniques may perform better in some cases and have advantages in practical applications.
2. **Multidimensional evaluation**: In addition to detection accuracy, time performance and sensitivity to hyper - parameter tuning are also important considerations for choosing the appropriate technique.
3. **Cover semi - supervised learning**: Considering the scarcity of abnormal samples in many log data, semi - supervised learning techniques should also receive sufficient attention and evaluation.

### Experimental design
The author selected six benchmark datasets and evaluated the following techniques:
- Supervised traditional techniques: Support Vector Machine (SVM), Random Forest (RF)
- Supervised deep - learning techniques: Long - Short - Term Memory Network (LSTM), LogRobust
- Semi - supervised traditional techniques: One - Class Support Vector Machine (OC - SVM)
- Semi - supervised deep - learning techniques: DeepLog

### Main findings
The experimental results show that:
- Supervised traditional and deep - learning techniques perform similarly in terms of detection accuracy and prediction time.
- Supervised traditional machine - learning techniques are less sensitive to hyper - parameter tuning.
- Semi - supervised techniques (both traditional and deep - learning) are inferior to supervised techniques in terms of detection accuracy.

### Conclusion
Although deep - learning techniques have received extensive attention in the LAD field, traditional machine - learning techniques (such as Random Forest) may be more suitable as choices for practical applications under multiple evaluation criteria. Semi - supervised techniques currently face challenges in terms of detection accuracy, and it is also a problem to collect enough abnormal log data.

In summary, this paper provides more powerful and comprehensive evidence regarding the relative advantages and disadvantages of different types of machine - learning techniques in LAD tasks through systematic empirical research.