Abstract:In recent years, with the increase in the scale and complexity of system software, how to effectively capture, analyze and locate the abnormal behavior generated during system operation has increasingly become a recognized problem in the software testing field. Traditional white-box-based anomaly detection methods need to provide system source code and cannot effectively detect abnormal behaviors that occur when the program is running. Black-box-based anomaly detection methods rely on test cases and have low code coverage. Anomaly detection based on the logs generated during system operation can alleviate the abovementioned problems. However, the existing system log-based abnormality detection method mainly extracts log template characteristics, and cannot effectively determine the logical and temporal abnormalities related to the log. Therefore, in order to detect anomalies in the log sequence more comprehensively, this paper proposes an anomaly detection method based on BiLSTM. This method combines the semantic and temporal characteristics of the log for modeling. In terms of semantics, the Bert natural language processing model is used to extract the contextual semantic information of the logs; in terms of time, the log time interval characteristics are extracted based on the three potential relationships of the logs. In addition, the proposed method also adds an attention mechanism to balance feature weights. Finally, we evaluate our method in experiments on two real datasets, HDFS and OpenStack. The experimental results show that our method has a great improvement in accuracy and recall.

LogCSS: Log anomaly detection based on BERT-CNN with context-semantics-statistics features

Natural Language Processing-based Model for Log Anomaly Detection

Log Anomaly Detection method based on BERT model optimization

SemLog: A Semantics-based Approach for Anomaly Detection in Big Data System Logs

BERT-Log: Anomaly Detection for System Logs Based on Pre-trained Language Model

CLDTLog: System Log Anomaly Detection Method Based on Contrastive Learning and Dual Objective Tasks

An Anomaly Detection Approach of Part-of-Speech Log Sequence Via Population Based Training

A Robust Log Classification Approach Based on Natural Language Processing

LogBERT: Log Anomaly Detection via BERT

Log anomaly detection method based on CNN and LSTM fusion

Semi-supervised log anomaly detection based on bidirectional temporal convolution network

LogCAE: an Approach for Log-based Anomaly Detection with Active Learning and Contrastive Learning

Research on Log Anomaly Detection Based on Sentence-BERT

Improving Performance of Log Anomaly Detection with Semantic and Time Features Based on BiLSTM-Attention

LightLog: A Lightweight Temporal Convolutional Network for Log Anomaly Detection on the Edge

LogPS: A Robust Log Sequential Anomaly Detection Approach Based on Natural Language Processing

PoSBert: Log Classification Via Modified Bert Based on Part-of-Speech Weight

Log-based Anomaly Detection Without Log Parsing

A Robust Wide & Deep Learning Framework for Log-Based Anomaly Detection

LogBD: A Log Anomaly Detection Method Based on Pretrained Models and Domain Adaptation

LLMeLog: an Approach for Anomaly Detection Based on LLM-enriched Log Events