Abstract:In recent years, with the increase in the scale and complexity of system software, how to effectively capture, analyze and locate the abnormal behavior generated during system operation has increasingly become a recognized problem in the software testing field. Traditional white-box-based anomaly detection methods need to provide system source code and cannot effectively detect abnormal behaviors that occur when the program is running. Black-box-based anomaly detection methods rely on test cases and have low code coverage. Anomaly detection based on the logs generated during system operation can alleviate the abovementioned problems. However, the existing system log-based abnormality detection method mainly extracts log template characteristics, and cannot effectively determine the logical and temporal abnormalities related to the log. Therefore, in order to detect anomalies in the log sequence more comprehensively, this paper proposes an anomaly detection method based on BiLSTM. This method combines the semantic and temporal characteristics of the log for modeling. In terms of semantics, the Bert natural language processing model is used to extract the contextual semantic information of the logs; in terms of time, the log time interval characteristics are extracted based on the three potential relationships of the logs. In addition, the proposed method also adds an attention mechanism to balance feature weights. Finally, we evaluate our method in experiments on two real datasets, HDFS and OpenStack. The experimental results show that our method has a great improvement in accuracy and recall.

Anomaly Detection in Unstructured Logs Using Attention-based Bi-LSTM Network

Research on System Log Anomaly Detection Combining Two-way Slice GRU and GA-Attention Mechanism

Improving Performance of Log Anomaly Detection with Semantic and Time Features Based on BiLSTM-Attention

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

Natural Language Processing-based Model for Log Anomaly Detection

An Anomaly Detection Approach of Part-of-Speech Log Sequence Via Population Based Training

Log-Based Anomaly Detection with Multi-Head Scaled Dot-Product Attention Mechanism

Unsupervised Anomaly Detection Based on System Logs.

An Anomaly Detection Algorithm for Logs Based on Self-attention Mechanism and BiGRU Model

Anomaly Detection Model for Log Based on LSTM Network and Variational Autoencoder

Robust Log-Based Anomaly Detection on Unstable Log Data

LogPS: A Robust Log Sequential Anomaly Detection Approach Based on Natural Language Processing

DualAttlog: Context aware dual attention networks for log-based anomaly detection

A LSTM-Based Anomaly Detection Model for Log Analysis

Machine Learning Based Anomaly Detection of Log Files Using Ensemble Learning and Self-Attention

Detecting Log Anomalies with Multi-Head Attention (LAMA)

A Survey of Deep Anomaly Detection for System Logs

Log-based Anomaly Detection Without Log Parsing

Anomaly Detection Mechanism Based on Hierarchical Weights Through Large-Scale Log Data

ConAnomaly: Content-Based Anomaly Detection for System Logs

Efficient Log Anomaly Detection Based on Dimension Reduction and Attention Aware TCN