Abstract:Numerous Deep Learning (DL)-based approaches have gained attention in software Log Anomaly Detection (LAD), yet class imbalance in training data remains a challenge, with anomalies often comprising less than 1% of datasets like Thunderbird. Existing DLLAD methods may underperform in severely imbalanced datasets. Although data resampling has proven effective in other software engineering tasks, it has not been explored in LAD. This study aims to fill this gap by providing an in-depth analysis of the impact of diverse data resampling methods on existing DLLAD approaches from two distinct perspectives. Firstly, we assess the performance of these DLLAD approaches across four datasets with different levels of class imbalance, and we explore the impact of resampling ratios of normal to abnormal data on DLLAD approaches. Secondly, we evaluate the effectiveness of the data resampling methods when utilizing optimal resampling ratios of normal to abnormal data. Our findings indicate that oversampling methods generally outperform undersampling and hybrid sampling methods. Data resampling on raw data yields superior results compared to data resampling in the feature space. These improvements are attributed to the increased attention given to important tokens. By exploring the resampling ratio of normal to abnormal data, we suggest generating more data for minority classes through oversampling while removing less data from majority classes through undersampling. In conclusion, our study provides valuable insights into the intricate relationship between data resampling methods and DLLAD. By addressing the challenge of class imbalance, researchers and practitioners can enhance DLLAD performance.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the **class - imbalance problem** that deep learning (DL) faces in log anomaly detection (LAD). Specifically, this research focuses on: 1. **The impact of class - imbalance on the performance of existing DL - LAD methods**: - In real - world datasets, the proportion of normal data far exceeds that of abnormal data. For example, in the Thunderbird dataset, abnormal data only accounts for 0.16% to 0.35%. Such severe class - imbalance can lead to poor performance of existing DL - LAD models in detecting anomalies, manifested as low precision or low recall. - A low recall means that some anomalies may be missed, while low precision will generate a large number of false positives, resulting in alert fatigue and resource waste. 2. **The application effect of data resampling methods in DL - LAD**: - Although data resampling has been proven effective in other software engineering tasks, it has not been fully explored in the field of log anomaly detection. Therefore, this research fills this research gap by evaluating the impact of multiple data resampling methods (including oversampling, undersampling, and hybrid sampling) on DL - LAD methods. - The researchers hope that through this work, they can provide valuable insights for researchers and practitioners to improve the performance of DL - LAD on class - imbalanced datasets. ### Specific objectives of the research - **Evaluate the performance of existing DL - LAD methods under different degrees of class - imbalance**: By testing existing methods on four datasets with different degrees of class - imbalance, understand the impact of class - imbalance on the performance of these methods. - **Explore the impact of the resampling ratio of normal and abnormal data on the performance of DL - LAD methods**: By adjusting the resampling ratio of normal and abnormal data, study the impact of different ratios on the performance of DL - LAD methods. - **Evaluate whether data resampling can improve the effectiveness of existing DL - LAD methods**: By using the optimal resampling ratio, evaluate the improvement effect of data resampling on the performance of DL - LAD methods. ### Main contributions 1. **Systematically evaluate the impact of data resampling methods on DL - LAD performance**: This research is the first to conduct a wide - ranging evaluation of the impact of data resampling methods in DL - LAD, covering 5,580 experiments and providing comprehensive evaluation and statistical analysis. 2. **Provide practical suggestions for researchers and practitioners**: The research results show that oversampling methods are generally superior to undersampling and hybrid sampling methods, especially simple methods directly applied to the original data (such as random oversampling). The research also recommends using methods such as ROS in datasets with severe class - imbalance. Through these studies, the paper hopes to provide effective solutions to the class - imbalance problem in DL - LAD and provide guidance for future research and practice.

On the Influence of Data Resampling for Deep Learning-Based Log Anomaly Detection: Insights and Recommendations

Towards Deeper Insights into Deep Learning from Imbalanced Data.

Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection

Deep Learning for Anomaly Detection in Log Data: A Survey

Try with Simpler -- An Evaluation of Improved Principal Component Analysis in Log-based Anomaly Detection

Does data sampling improve deep learning-based vulnerability detection? Yeas! and Nays!

Value-Aware Resampling and Loss for Imbalanced Classification

OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning

ADA: Adaptive Deep Log Anomaly Detector

Semi-supervised learning via DQN for log anomaly detection

Resampling imbalanced data for network intrusion detection datasets

Fair Anomaly Detection For Imbalanced Groups

On the Effectiveness of Log Representation for Log-based Anomaly Detection

A Deep Learning Model for Network Intrusion Detection with Imbalanced Data

Reducing Events to Augment Log-based Anomaly Detection Models: An Empirical Study

Effect Improved for High-Dimensional and Unbalanced Data Anomaly Detection Model Based on KNN-SMOTE-LSTM

Training Data Debugging for the Fairness of Machine Learning Software

AFALog: A General Augmentation Framework for Log-based Anomaly Detection with Active Learning

A Comprehensive Study of Machine Learning Techniques for Log-Based Anomaly Detection

LLMeLog: an Approach for Anomaly Detection Based on LLM-enriched Log Events

DeepUserLog: Deep Anomaly Detection on User Log Using Semantic Analysis and Key-Value Data