Abstract:Log analysis is one of the main techniques that engineers use for troubleshooting large-scale software systems. Over the years, many supervised, semi-supervised, and unsupervised log analysis methods have been proposed to detect system anomalies by analyzing system logs. Among these, semi-supervised methods have garnered increasing attention as they strike a balance between relaxed labeled data requirements and optimal detection performance, contrasting with their supervised and unsupervised counterparts. However, existing semi-supervised methods overlook the potential bias introduced by highly frequent log messages on the learned normal patterns, which leads to their less than satisfactory performance. In this study, we propose LogSD, a novel semi-supervised self-supervised learning approach. LogSD employs a dual-network architecture and incorporates a frequency-based masking scheme, a global-to-local reconstruction paradigm and three self-supervised learning tasks. These features enable LogSD to focus more on relatively infrequent log messages, thereby effectively learning less biased and more discriminative patterns from historical normal data. This emphasis ultimately leads to improved anomaly detection performance. Extensive experiments have been conducted on three commonly-used datasets and the results show that LogSD significantly outperforms eight state-of-the-art benchmark methods.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the bias problem existing in the existing semi - supervised log anomaly detection methods. That is, these methods are too biased towards frequently - occurring log messages when learning normal patterns, while ignoring low - frequency events. This bias makes it difficult for the model to distinguish between instances containing low - frequency normal events and abnormal events, thus affecting the performance of anomaly detection.
### Specific Problem Description
1. **Limitations of Existing Methods**:
- Semi - supervised methods are easily influenced by high - frequency log messages when learning normal patterns, causing the model to over - fit high - frequency events.
- Low - frequency events are ignored or underestimated, making it difficult for the model to capture all types of normal behavior patterns.
- This bias will lead to a decline in anomaly detection performance in practical applications, especially when dealing with large - scale and complex system logs.
2. **Specific Application Scenarios**:
- Modern software systems are large - scale and complex, generating a large amount of log data.
- Manual analysis of these logs to find anomalies has become impractical, so automated log anomaly detection has become crucial.
- Existing supervised, unsupervised, and semi - supervised methods have their own advantages and disadvantages, but none of them have completely solved the above - mentioned bias problem.
### Solution Overview
To solve these problems, the author proposes a new method named LogSD (Log - based Self - supervised Detection). LogSD improves the existing semi - supervised log anomaly detection methods in the following aspects:
1. **Frequency - based Masking Scheme**:
- Use the occurrence frequency of events to guide the masking process of sequences, and divide events into high - frequency and low - frequency two categories.
- By dynamically adjusting the masking ratio, ensure that the model can pay attention to low - frequency events, thereby reducing bias.
2. **Global - to - Local Reconstruction Paradigm**:
- Use global input for encoding, but only use local (low - frequency) events when calculating the reconstruction loss.
- This design forces the model to focus on reconstructing low - frequency events rather than all events, so as to better learn normal patterns.
3. **Dual - Network Architecture and Self - supervised Tasks**:
- Use an auto - encoder sub - network (AE) and an encoder - only sub - network (EO), and introduce three self - supervised tasks: reconstruction task, one - class classification task, and prediction task.
- The AE sub - network focuses on reconstructing low - frequency events, while the EO sub - network learns the representation of high - frequency events in the absence of low - frequency events.
Through these improvements, LogSD can learn normal patterns more evenly during the training process, avoid excessive dependence on high - frequency events, and thus improve the accuracy and robustness of anomaly detection. Experimental results show that LogSD significantly outperforms eight other latest benchmark methods on multiple commonly - used data sets.