Abstract:Log lines consist of static parts that characterize their structure and enable assignment of event types, and event parameters, i.e., variable parts that provide specific information on system processes, such as host and user names, IP addresses, and file operations. Many detection approaches only focus on anomalous event type occurrences, i.e., they parse log lines to derive unique event identifiers and subsequently detect anomalies in event sequences or event count vectors, but neglect variable parts of log lines entirely during analysis. This is especially problematic, when monitoring strongly structured log data that contains only a small number of distinct event types, for example, logs that consist of strict key value pairs, i.e., parameters that occur consistently throughout all log lines, such as it is case in access and audit logs. Thus, novel approaches are required, which focus on analysis of log lines' variable parts. In this paper, we propose the variable type detector (VTD), a novel unsupervised approach that autonomously analyzes variable log line parts to enable anomaly detection. It assigns data types to each variable, which also include probability distributions for discrete and continuous variables. The VTD raises an alarm if a variable's data type changes. Furthermore, it implements a robust indicator function that reduces false positives by tracking the data type history of each variable and reports only significant data type changes. Additionally, an event indicator enables event-based anomaly detection by taking into account the data types of all variables of a single event type. The evaluation conducted on open-source log data, demonstrates the effectiveness of the VTD compared to conventional anomaly detection approaches, such as time series analysis and PCA. Consequently, the VTD acts as a solution that extends the intrusion detection capabilities of security information and event management (SIEM) and integrates with modern concepts of endpoint detection and response (EDR) and extended detection and responses (XDR), while simultaneously serving as an asset for process monitoring that supports user and entity behavior analytics (UEBA).

TeleDAL: a regression-based template-less unsupervised method for finding anomalies in log sequences

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

Natural Language Processing-based Model for Log Anomaly Detection

Log-based Anomaly Detection Without Log Parsing

Deep Learning for Anomaly Detection in Log Data: A Survey

Log Sequence Anomaly Detection Based on Local Information Extraction and Globally Sparse Transformer Model

LogELECTRA: Self-supervised Anomaly Detection for Unstructured Logs

HitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log

Log-based Anomaly Detection of Enterprise Software: An Empirical Study

OneLog: Towards End-to-End Training in Software Log Anomaly Detection

MLog: Mogrifier LSTM-based Log Anomaly Detection Approach Using Semantic Representation

Log-based Anomaly Detection with Deep Learning: How Far Are We?

TPLogAD: Unsupervised Log Anomaly Detection Based on Event Templates and Key Parameters

LogGD:Detecting Anomalies from System Logs by Graph Neural Networks

Log-based Anomaly Detection based on EVT Theory with feedback

Anomaly Detection Model for Log Based on LSTM Network and Variational Autoencoder

LogAnMeta: Log Anomaly Detection Using Meta Learning

Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models

LogGPT: Log Anomaly Detection via GPT

Analysis of Statistical Properties of Variables in Log Data for Advanced Anomaly Detection in Cyber Security

ADA: Adaptive Deep Log Anomaly Detector