Abstract:In the era of rapid Internet development, log data has become indispensable for recording the operations of computer devices and software. These data provide valuable insights into system behavior and necessitate thorough analysis. Recent advances in text analysis have enabled deep learning to achieve significant breakthroughs in log anomaly detection. However, the high cost of manual annotation and the dynamic nature of usage scenarios present major challenges to effective log analysis. This study proposes a novel log feature extraction model called DualGCN-LogAE, designed to adapt to various scenarios. It leverages the expressive power of large models for log content analysis and the capability of graph structures to encapsulate correlations between logs. It retains key log information while integrating the causal relationships between logs to achieve effective feature extraction. Additionally, we introduce Log2graphs, an unsupervised log anomaly detection method based on the feature extractor. By employing graph clustering algorithms for log anomaly detection, Log2graphs enables the identification of abnormal logs without the need for labeled data. We comprehensively evaluate the feature extraction capability of DualGCN-LogAE and the anomaly detection performance of Log2graphs using public log datasets across five different scenarios. Our evaluation metrics include detection accuracy and graph clustering quality scores. Experimental results demonstrate that the log features extracted by DualGCN-LogAE outperform those obtained by other methods on classic classifiers. Moreover, Log2graphs surpasses existing unsupervised log detection methods, providing a robust tool for advancing log anomaly detection research.

Log-Based Anomaly Detection with Multi-Head Scaled Dot-Product Attention Mechanism

Research on System Log Anomaly Detection Combining Two-way Slice GRU and GA-Attention Mechanism

Anomaly Detection in Unstructured Logs Using Attention-based Bi-LSTM Network

DualAttlog: Context aware dual attention networks for log-based anomaly detection

LogAttn: Unsupervised Log Anomaly Detection with an AutoEncoder Based Attention Mechanism

Improving Performance of Log Anomaly Detection with Semantic and Time Features Based on BiLSTM-Attention

Detecting Log Anomalies with Multi-Head Attention (LAMA)

Natural Language Processing-based Model for Log Anomaly Detection

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

An Anomaly Detection Algorithm for Logs Based on Self-attention Mechanism and BiGRU Model

Machine Learning Based Anomaly Detection of Log Files Using Ensemble Learning and Self-Attention

An Anomaly Detection Approach of Part-of-Speech Log Sequence Via Population Based Training

Efficient Log Anomaly Detection Based on Dimension Reduction and Attention Aware TCN

Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection

Improving Log-Based Anomaly Detection with Component-Aware Analysis

A Survey of Deep Anomaly Detection for System Logs

Distributed system anomaly detection using deep learning‐based log analysis

LogPS: A Robust Log Sequential Anomaly Detection Approach Based on Natural Language Processing

Robust Log-Based Anomaly Detection on Unstable Log Data

Anomaly Detection Mechanism Based on Hierarchical Weights Through Large-Scale Log Data

Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction