Abstract:Introduction: Log anomaly detection is essential for monitoring and maintaining the normal operation of systems. With the rapid development and maturation of deep learning technologies, deep learning-based log anomaly detection has become a prominent research area. However, existing methods primarily concentrate on directly detecting log data in a single stage using specific anomaly information, such as log sequential information or log semantic information. This leads to a limited understanding of log data, resulting in low detection accuracy and poor model robustness. Methods: To tackle this challenge, we propose LogMS, a multi-stage log anomaly detection method based on multi-source information fusion and probability label estimation. Before anomaly detection, the logs undergo parsing and vectorization to capture semantic information. Subsequently, we propose a multi-source information fusion-based long short-term memory (MSIF-LSTM) network for the initial stage of anomaly log detection. By fusing semantic information, sequential information, and quantitative information, MSIF-LSTM enhances the anomaly detection capability. Furthermore, we introduce a probability label estimation-based gate recurrent unit (PLE-GRU) network, which leverages easily obtainable normal log labels to construct pseudo-labeled data and train a GRU for further detection. PLE-GRU enhances the detection capability from the perspective of label information. To ensure the overall efficiency of the LogMS, the second-stage will only be activated when anomalies are not detected in the first stage. Results and Discussion: Experimental results demonstrate that LogMS outperforms baseline models across various log anomaly detection datasets, exhibiting superior performance in robustness testing.

MLog: Mogrifier LSTM-based Log Anomaly Detection Approach Using Semantic Representation

An Anomaly Detection Approach of Part-of-Speech Log Sequence Via Population Based Training

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

LLMeLog: an Approach for Anomaly Detection Based on LLM-enriched Log Events

Log-based Anomaly Detection Without Log Parsing

OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning

Natural Language Processing-based Model for Log Anomaly Detection

Log anomaly detection method based on CNN and LSTM fusion

Anomaly Detection Model for Log Based on LSTM Network and Variational Autoencoder

LogPS: A Robust Log Sequential Anomaly Detection Approach Based on Natural Language Processing

LogMS: a multi-stage log anomaly detection method based on multi-source information fusion and probability label estimation

A LSTM-Based Anomaly Detection Model for Log Analysis

Leveraging RAG-Enhanced Large Language Model for Semi-Supervised Log Anomaly Detection

LogLLM: Log-based Anomaly Detection Using Large Language Models

OneLog: towards end-to-end software log anomaly detection

Detecting Anomalies in Software Execution Logs with Siamese Network

CausalConvLSTM: Semi-Supervised Log Anomaly Detection Through Sequence Modeling

Robust Log-Based Anomaly Detection on Unstable Log Data

OneLog: Towards End-to-End Training in Software Log Anomaly Detection

ConAnomaly: Content-Based Anomaly Detection for System Logs

MDFULog: Multi-Feature Deep Fusion of Unstable Log Anomaly Detection Model