Abstract:In the era of rapid Internet development, log data has become indispensable for recording the operations of computer devices and software. These data provide valuable insights into system behavior and necessitate thorough analysis. Recent advances in text analysis have enabled deep learning to achieve significant breakthroughs in log anomaly detection. However, the high cost of manual annotation and the dynamic nature of usage scenarios present major challenges to effective log analysis. This study proposes a novel log feature extraction model called DualGCN-LogAE, designed to adapt to various scenarios. It leverages the expressive power of large models for log content analysis and the capability of graph structures to encapsulate correlations between logs. It retains key log information while integrating the causal relationships between logs to achieve effective feature extraction. Additionally, we introduce Log2graphs, an unsupervised log anomaly detection method based on the feature extractor. By employing graph clustering algorithms for log anomaly detection, Log2graphs enables the identification of abnormal logs without the need for labeled data. We comprehensively evaluate the feature extraction capability of DualGCN-LogAE and the anomaly detection performance of Log2graphs using public log datasets across five different scenarios. Our evaluation metrics include detection accuracy and graph clustering quality scores. Experimental results demonstrate that the log features extracted by DualGCN-LogAE outperform those obtained by other methods on classic classifiers. Moreover, Log2graphs surpasses existing unsupervised log detection methods, providing a robust tool for advancing log anomaly detection research.

An Effective Feature Representation of Web Log Data by Leveraging Byte Pair Encoding and TF-IDF

A Semantic-aware Representation Framework for Online Log Analysis

Assessing the impact of bag‐of‐words versus word‐to‐vector embedding methods and dimension reduction on anomaly detection from log files

Pattern-Based Deep Learning Method to Extract Information from the Log Dataset.

Log2vec: A Heterogeneous Graph Embedding Based Approach for Detecting Cyber Threats within Enterprise

Matrix Dimensionality Reduction for Mining Web Logs

On the Effectiveness of Log Representation for Log-based Anomaly Detection

Exploiting Meta Features for Dependency Parsing and Part-Of-Speech Tagging

MDFULog: Multi-Feature Deep Fusion of Unstable Log Anomaly Detection Model

A Robust Wide & Deep Learning Framework for Log-Based Anomaly Detection

A Robust Log Classification Approach Based on Natural Language Processing

Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction

Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems

A Method to Automatically Filter Log Evidences for Intrusion Forensics

Latent Text Mining for Cybercrime Forensics

End-To-End Anomaly Detection for Identifying Malicious Cyber Behavior through NLP-Based Log Embeddings

Deep Forest with LRRS Feature for Fine-grained Website Fingerprinting with Encrypted SSL/TLS

FSMFLog: Discovering Anomalous Logs Combining Full Semantic Information and Multifeature Fusion

Biglog: Unsupervised Large-scale Pre-training for a Unified Log Representation

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things

Learning Representations for Log Data in Cybersecurity