Abstract:Log lines consist of static parts that characterize their structure and enable assignment of event types, and event parameters, i.e., variable parts that provide specific information on system processes, such as host and user names, IP addresses, and file operations. Many detection approaches only focus on anomalous event type occurrences, i.e., they parse log lines to derive unique event identifiers and subsequently detect anomalies in event sequences or event count vectors, but neglect variable parts of log lines entirely during analysis. This is especially problematic, when monitoring strongly structured log data that contains only a small number of distinct event types, for example, logs that consist of strict key value pairs, i.e., parameters that occur consistently throughout all log lines, such as it is case in access and audit logs. Thus, novel approaches are required, which focus on analysis of log lines' variable parts. In this paper, we propose the variable type detector (VTD), a novel unsupervised approach that autonomously analyzes variable log line parts to enable anomaly detection. It assigns data types to each variable, which also include probability distributions for discrete and continuous variables. The VTD raises an alarm if a variable's data type changes. Furthermore, it implements a robust indicator function that reduces false positives by tracking the data type history of each variable and reports only significant data type changes. Additionally, an event indicator enables event-based anomaly detection by taking into account the data types of all variables of a single event type. The evaluation conducted on open-source log data, demonstrates the effectiveness of the VTD compared to conventional anomaly detection approaches, such as time series analysis and PCA. Consequently, the VTD acts as a solution that extends the intrusion detection capabilities of security information and event management (SIEM) and integrates with modern concepts of endpoint detection and response (EDR) and extended detection and responses (XDR), while simultaneously serving as an asset for process monitoring that supports user and entity behavior analytics (UEBA).

System log clustering approaches for cyber security applications: A survey

Dynamic log file analysis: An unsupervised cluster evolution approach for anomaly detection

Algorithms and methods of data clustering in the analysis of information security event logs

ClusterLog: Clustering Logs for Effective Log-based Anomaly Detection

INVESTIGATION OF CLUSTERING AND CLASSIFICATION METHODS FOR INTELLECTUAL ANALYSIS OF LOG FILES

Applying High-Performance Bioinformatics Tools for Outlier Detection in Log Data

Log clustering based problem identification for online service systems

Online Log Data Analysis With Efficient Machine Learning: A Review

Deep Learning for Anomaly Detection in Log Data: A Survey

Log Analysis Techniques using Clustering in Network Forensics

Analysis of Statistical Properties of Variables in Log Data for Advanced Anomaly Detection in Cyber Security

A comprehensive survey of clustering algorithms: State-of-the-art machine learning applications, taxonomy, challenges, and future research prospects

Error Log Clustering of Internet Software

A Log Analysis Audit Model Based on Optimized Clustering Algorithm

Discovering Insider Threats from Log Data with High-Performance Bioinformatics Tools

An Intelligent Framework for Log Anomaly Detection Based on Log Template Extraction

ThreatCluster: Threat Clustering for Information Overload Reduction in Computer Emergency Response Teams

Comprehensive Analysis and Evaluation of Anomalous User Activity in Web Server Logs

An ensemble-based framework for user behaviour anomaly detection and classification for cybersecurity

Detecting and Identifying Insider Threats Based on Advanced Clustering Methods

An Integrated Method for Anomaly Detection From Massive System Logs.