Abstract:Logs are imperative in the maintenance of online service systems, which often encompass important information for effective failure mitigation. While existing anomaly detection methodologies facilitate the identification of anomalous logs within extensive runtime data, manual investigation of log messages by engineers remains essential to comprehend faults, which is labor-intensive and error-prone. Upon examining the log-based troubleshooting practices at CloudA, we find that engineers typically prioritize two categories of log information for diagnosis. These include fault-indicating descriptions, which record abnormal system events, and fault-indicating parameters, which specify the associated entities. Motivated by this finding, we propose an approach to automatically extract such faultindicating information from logs for fault diagnosis, named LoFI. LoFI comprises two key stages. In the first stage, LoFI performs coarse-grained filtering to collect logs related to the faults based on semantic similarity. In the second stage, LoFI leverages a pre-trained language model with a novel prompt-based tuning method to extract fine-grained information of interest from the collected logs. We evaluate LoFI on logs collected from Apache Spark and an industrial dataset from CloudA. The experimental results demonstrate that LoFI outperforms all baseline methods by a significant margin, achieving an absolute improvement of 25.8~37.9 in F1 over the best baseline method, ChatGPT. This highlights the effectiveness of LoFI in recognizing fault-indicating information. Furthermore, the successful deployment of LoFI at CloudA and user studies validate the utility of our method. The code and data are available at <a class="link-external link-https" href="https://github.com/Jun-jie-Huang/LoFI" rel="external noopener nofollow">this https URL</a>.

Log-Based OpenStack Fault Diagnosis by Machine Learning

Fault Diagnosis Method Based on Improved Deep Boltzmann Machines

Demystifying and Extracting Fault-indicating Information from Logs for Failure Diagnosis

An Approach to Pinpointing Bug-Induced Failure in Logs of Open Cloud Platforms

LogKG: Log Failure Diagnosis through Knowledge Graph

Machine Deserves Better Logging: A Log Enhancement Approach for Automatic Fault Diagnosis

Weighted Time Series Fault Diagnosis Based on a Stacked Sparse Autoencoder

Brief Announcement: Automatic Log Enhancement for Fault Diagnosis.

Research on anomaly detection and real-time reliability evaluation with the log of cloud platform

Enhancing the Analysis of Software Failures in Cloud Computing Systems with Deep Learning

Fault diagnosis based on deep learning by extracting inherent common feature of multi-source heterogeneous data

A Grid Fault Diagnosis Method Based on Stacking Algorithm

Deep Learning-Based Composite Fault Diagnosis

Online Fault Classification in HPC Systems Through Machine Learning

Fault diagnosis of key components in the rotating machinery based on Fourier transform multi-filter decomposition and optimized LightGBM

Application of local outlier factor method and back-propagation neural network for steel plates fault diagnosis

Multi-sample-distances-fusion- and generalized-Pareto-distribution-based open-set fault diagnosis of rolling bearing

Automatic fault diagnosis of rolling bearings under multiple working conditions based on unsupervised stack denoising autoencoder

Fault Diagnosis Technology for Complex Mechanical Systems Based on Deep Learning

Anomaly detecting and ranking of the cloud computing platform by multi-view learning

Bearing fault diagnosis base on multi-scale CNN and LSTM model