Abstract:Host logs, in particular, Windows Event Logs, are a valuable source of information often collected by security operation centers (SOCs). The semi-structured nature of host logs inhibits automated analytics, and while manual analysis is common, the sheer volume makes manual inspection of all logs impossible. Although many powerful algorithms for analyzing time-series and sequential data exist, utilization of such algorithms for most cyber security applications is either infeasible or requires tailored, research-intensive preparations. In particular, basic mathematic and algorithmic developments for providing a generalized, meaningful similarity metric on system logs is needed to bridge the gap between many existing sequential data mining methods and this currently available but under-utilized data source. In this paper, we provide a rigorous definition of a metric product space on Windows Event Logs, providing an embedding that allows for the application of established machine learning and time-series analysis methods. We then demonstrate the utility and flexibility of this embedding with multiple use-cases on real data: (1) comparing known infected to new host log streams for attack detection and forensics, (2) collapsing similar streams of logs into semantically-meaningful groups (by user, by role), thereby reducing the quantity of data but not the content, (3) clustering logs as well as short sequences of logs to identify and visualize user behaviors and background processes over time. Overall, we provide a metric space framework for general host logs and log sequences that respects semantic similarity and facilitates a wide variety of data science analytics to these logs without data-specific preparations for each.

A Flexible Architecture for Statistical Learning and Data Mining from System Log Streams

Socialanalysis: A Real-Time Query And Mining System From Social Media Data Streams

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

Online System Problem Detection by Mining Patterns of Console Logs

Detecting Large-Scale System Problems by Mining Console Logs

System Problem Detection by Mining Console Logs - Escholarship

Mining Console Logs for Large-Scale System Problem Detection.

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

A Big Data Architecture for Log Data Storage and Analysis

Olaping stream data: Is it feasible

Design and Implementation of Log Data Analysis Management System Based on Hadoop

A Framework for Pattern Mining and Anomaly Detection in Multi-dimensional Time Series and Event Logs

LogFlash: Real-time Streaming Anomaly Detection and Diagnosis from System Logs for Large-scale Software Systems

A Large-scale Distributed Video Parsing and Evaluation Platform

An Integrated Method for Anomaly Detection From Massive System Logs.

A Distributed Text Mining System for Online Web Textual Data Analysis

Defining a Metric Space of Host Logs and Operational Use Cases

Finding Needles in the Haystack: Harnessing Syslogs for Data Center Management

Log analysis in cloud computing environment with Hadoop and Spark

Analyzing Logs of Large-Scale Software Systems using Time Curves Visualization

A Scalable Data Mining Architecture for Bioinformation