Abstract:Goal: We consider the problem of automatically grouping logs of runs that failed for the same underlying reasons, so that they can be treated more effectively, and investigate the following questions: (1) Does an approach developed to identify problems in system logs generalize to identifying problems in continuous deployment logs? (2) How does dimensionality reduction affect the quality of automated log clustering? (3) How does the criterion used for merging clusters in the clustering algorithm affect clustering quality? Method: We replicate and extend earlier work on clustering system log files to assess its generalization to continuous deployment logs. We consider the optional inclusion of one of these dimensionality reduction techniques: Principal Component Analysis (PCA), Latent Semantic Indexing (LSI), and Non-negative Matrix Factorization (NMF). Moreover, we consider three alternative cluster merge criteria (Single Linkage, Average Linkage, and Weighted Linkage), in addition to the Complete Linkage criterion used in earlier work. We empirically evaluate the 16 resulting configurations on continuous deployment logs provided by our industrial collaborator. Results: Our study shows that (1) identifying problems in continuous deployment logs via clustering is feasible, (2) including NMF significantly improves overall accuracy and robustness, and (3) Complete Linkage performs best of all merge criteria analyzed. Conclusions: We conclude that problem identification via automated log clustering is improved by including dimensionality reduction, as it decreases the pipeline's sensitivity to parameter choice, thereby increasing its robustness for handling different inputs.

Multidimensional Analysis of System Logs in Large-scale Cluster Systems

A Distributed Data Mining System Framework for Mobile Internet Access Log Based on Hadoop.

A multivariable statistical process monitoring method based on multiscale analysis

Big-data Analysis of Multi-Source Logs for Anomaly Detection on Network-Based System.

LogMaster: Mining Event Correlations in Logs of Large scale Cluster Systems

Multivariate Log-based Anomaly Detection for Distributed Database

An Approach to Failure Prediction in Cluster by Self-updating Cause-and-Effect Graph

Detecting Large-Scale System Problems by Mining Console Logs

Log clustering based problem identification for online service systems

Analyzing Logs of Large-Scale Software Systems using Time Curves Visualization

An Integrated Method for Anomaly Detection From Massive System Logs.

A Method of Large - Scale Log Pattern Mining

A Method of Failure Diagnosis Based on Association Rules and Study

Network Log Analysis with SQL-on-Hadoop

A parallel clustering algorithm for logs data based on Hadoop platform

Machine Deserves Better Logging: A Log Enhancement Approach for Automatic Fault Diagnosis

ClusterLog: Clustering Logs for Effective Log-based Anomaly Detection

Improving Problem Identification via Automated Log Clustering using Dimensionality Reduction

Mining Console Logs for Large-Scale System Problem Detection.

An empirical study of major page faults for failure diagnosis in cluster systems

Log analysis in cloud computing environment with Hadoop and Spark