Abstract:Introduction: Log anomaly detection is essential for monitoring and maintaining the normal operation of systems. With the rapid development and maturation of deep learning technologies, deep learning-based log anomaly detection has become a prominent research area. However, existing methods primarily concentrate on directly detecting log data in a single stage using specific anomaly information, such as log sequential information or log semantic information. This leads to a limited understanding of log data, resulting in low detection accuracy and poor model robustness. Methods: To tackle this challenge, we propose LogMS, a multi-stage log anomaly detection method based on multi-source information fusion and probability label estimation. Before anomaly detection, the logs undergo parsing and vectorization to capture semantic information. Subsequently, we propose a multi-source information fusion-based long short-term memory (MSIF-LSTM) network for the initial stage of anomaly log detection. By fusing semantic information, sequential information, and quantitative information, MSIF-LSTM enhances the anomaly detection capability. Furthermore, we introduce a probability label estimation-based gate recurrent unit (PLE-GRU) network, which leverages easily obtainable normal log labels to construct pseudo-labeled data and train a GRU for further detection. PLE-GRU enhances the detection capability from the perspective of label information. To ensure the overall efficiency of the LogMS, the second-stage will only be activated when anomalies are not detected in the first stage. Results and Discussion: Experimental results demonstrate that LogMS outperforms baseline models across various log anomaly detection datasets, exhibiting superior performance in robustness testing.

Detecting Log Anomalies with Multi-Head Attention (LAMA)

Research on System Log Anomaly Detection Combining Two-way Slice GRU and GA-Attention Mechanism

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

DualAttlog: Context aware dual attention networks for log-based anomaly detection

OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning

An Anomaly Detection Approach of Part-of-Speech Log Sequence Via Population Based Training

LogLLM: Log-based Anomaly Detection Using Large Language Models

Natural Language Processing-based Model for Log Anomaly Detection

MLog: Mogrifier LSTM-based Log Anomaly Detection Approach Using Semantic Representation

A LSTM-Based Anomaly Detection Model for Log Analysis

LogAttn: Unsupervised Log Anomaly Detection with an AutoEncoder Based Attention Mechanism

LLMeLog: an Approach for Anomaly Detection Based on LLM-enriched Log Events

LogPS: A Robust Log Sequential Anomaly Detection Approach Based on Natural Language Processing

Anomaly Detection Model for Log Based on LSTM Network and Variational Autoencoder

Attention Based CNN-LSTM Network for Anomaly Pattern Classification of Multivariate Time Series

Log Sequence Anomaly Detection Based on Local Information Extraction and Globally Sparse Transformer Model

LogPal: A Generic Anomaly Detection Scheme of Heterogeneous Logs for Network Systems

Leveraging RAG-Enhanced Large Language Model for Semi-Supervised Log Anomaly Detection

LogCAE: an Approach for Log-based Anomaly Detection with Active Learning and Contrastive Learning

LogMS: a multi-stage log anomaly detection method based on multi-source information fusion and probability label estimation

CausalConvLSTM: Semi-Supervised Log Anomaly Detection Through Sequence Modeling