Abstract:Modern software systems produce vast amounts of logs, serving as an essential resource for anomaly detection. Artificial Intelligence for IT Operations (AIOps) tools have been developed to automate the process of log-based anomaly detection for software systems. Three practical challenges are widely recognized in this field: high data labeling costs, evolving logs in dynamic systems, and adaptability across different systems. In this paper, we propose CroSysLog, an AIOps tool for log-event level anomaly detection, specifically designed in response to these challenges. Following prior approaches, CroSysLog uses a neural representation approach to gain a nuanced understanding of logs and generate representations for individual log events accordingly. CroSysLog can be trained on source systems with sufficient labeled logs from open datasets to achieve robustness, and then efficiently adapt to target systems with a few labeled log events for effective anomaly detection. We evaluate CroSysLog using open datasets of four large-scale distributed supercomputing systems: BGL, Thunderbird, Liberty, and Spirit. We used random log splits, maintaining the chronological order of consecutive log events, from these systems to train and evaluate CroSysLog. These splits were widely distributed across a one/two-year span of each system's log collection duration, thereby capturing the evolving nature of the logs in each system. Our results show that, after training CroSysLog on Liberty and BGL as source systems, CroSysLog can efficiently adapt to target systems Thunderbird and Spirit using a few labeled log events from each target system, effectively performing anomaly detection for these target systems. The results demonstrate that CroSysLog is a practical, scalable, and adaptable tool for log-event level anomaly detection in operational and maintenance contexts of software systems.

Feedback-Aware Anomaly Detection Through Logs for Large-Scale Software Systems

Augmenting Log-based Anomaly Detection Models to Reduce False Anomalies with Human Feedback

Log-based Anomaly Detection based on EVT Theory with feedback

Research on System Log Anomaly Detection Combining Two-way Slice GRU and GA-Attention Mechanism

Interactive Context-Aware Anomaly Detection Guided by User Feedback

Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection

Log-based Anomaly Detection of Enterprise Software: An Empirical Study

Natural Language Processing-based Model for Log Anomaly Detection

Improving Log-Based Anomaly Detection with Component-Aware Analysis

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

Hilogx: noise-aware log-based anomaly detection with human feedback

Log-Based Anomaly Detection with Multi-Head Scaled Dot-Product Attention Mechanism

Distributed system anomaly detection using deep learning‐based log analysis

An Integrated Method for Anomaly Detection From Massive System Logs.

Cross-System Software Log-based Anomaly Detection Using Meta-Learning

Machine Learning Based Anomaly Detection of Log Files Using Ensemble Learning and Self-Attention

Robust and Transferable Log-based Anomaly Detection.

Anomaly Detection in Unstructured Logs Using Attention-based Bi-LSTM Network

Robust and Rapid Adaption for Concept Drift in Software System Anomaly Detection.

An Empirical Investigation of Practical Log Anomaly Detection for Online Service Systems.

Brief Announcement: Automatic Log Enhancement for Fault Diagnosis.