Abstract:Log messages provide a valuable source of runtime information for ensuring the safety and consistency of systems. Recently, many machine learning and deep learning methods have been proposed to automatically detect anomalous log messages, obviating the need for manual detection by experts. However, we find that in practice, the effectiveness of existing learning-based methods is severely affected by incomplete information and distribution shift. Specifically, each log message can actually be parsed into a fixed number of key information fields, while existing methods analyze log messages using only the log event information and ignore other useful information fields that can be critical to anomaly detection. Further, the distribution of real-world log messages changes continuously due to the dynamic nature of the runtime environment and thus, a detection model conventionally trained based on the unrealistic i.i.d. assumption may not provide the expected and consistent performance. In this paper, we present a robust and transferable anomaly detection framework RT-Log to address the above problems. To perform a comprehensive analysis of log messages, we introduce an adaptive relation modeling technique, which captures feature interactions among log information fields selectively and dynamically for effective and interpretable log representations. To establish its robustness and transferability, we propose a general environment generalization technique for learning the environment invariant representations that can generalize across different runtime environments. We evaluate the anomaly detection performance of RT-Log on large real-world datasets. Extensive experimental results demonstrate that RT-Log consistently outperforms state-of-the-art methods by a significant margin under different settings.

Optimizing Anomaly Detection in Large-scale Logs

An Integrated Method for Anomaly Detection From Massive System Logs.

MoniLog: An Automated Log-Based Anomaly Detection System for Cloud Computing Infrastructures

Natural Language Processing-based Model for Log Anomaly Detection

A Robust Wide & Deep Learning Framework for Log-Based Anomaly Detection

A Hyperparameter Tuning Approach for an Online Log Parser

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

AIOps-Driven Enhancement of Log Anomaly Detection in Unsupervised Scenarios

Log-based Anomaly Detection based on EVT Theory with feedback

Log‐based anomaly detection for distributed systems: State of the art, industry experience, and open issues

Online System Problem Detection by Mining Patterns of Console Logs

ClusterLog: Clustering Logs for Effective Log-based Anomaly Detection

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

Execution Anomaly Detection in Large-Scale Systems Through Console Log Analysis

Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction

Exploring Optimal Transport for Event-Level Anomaly Detection at the Large Hadron Collider

Finding Gold in the Sand: Identifying Anomaly Indicators Though Huge Amount Security Logs

Robust and Transferable Log-based Anomaly Detection.

Distributed system anomaly detection using deep learning‐based log analysis

Reducing Events to Augment Log-based Anomaly Detection Models: An Empirical Study

Feedback-Aware Anomaly Detection Through Logs for Large-Scale Software Systems