Abstract:Many systems, such as distributed operating systems, complex networks, and high throughput web-based applications, are continuously generating large volume of event logs. These logs contain useful information to help system administrators to understand the system running status and to pinpoint the system failures. Generally, due to the scale and complexity of modern systems, the generated logs are beyond the analytic power of human beings. Therefore, it is imperative to develop a comprehensive log analysis system to support effective system management. Although a number of log mining techniques have been proposed to address specific log analysis use cases, few research and industrial efforts have been paid on providing integrated systems with an end-to-end solution to facilitate the log analysis routines. In this paper, we design and implement an integrated system, called FIU Log Analysis Platform (a.k.a. FLAP), that aims to facilitate the data analytics for system event logs. FLAP provides an end-to-end solution that utilizes advanced data mining techniques to assist log analysts to conveniently, timely, and accurately conduct event log knowledge discovery, system status investigation, and system failure diagnosis. Specifically, in FLAP, state-of-the-art template learning techniques are used to extract useful information from unstructured raw logs; advanced data transformation techniques are proposed and leveraged for event transformation and storage; effective event pattern mining, event summarization, event querying, and failure prediction techniques are designed and integrated for log analytics; and user-friendly interfaces are utilized to present the informative analysis results intuitively and vividly. Since 2016, FLAP has been used by Huawei Technologies Co. Ltd for internal event log analysis, and has provided effective support in its system operation and workflow optimization.

LogAssist: Assisting Log Analysis Through Log Summarization

Using Finite-State Models for Log Differencing

LogLM: From Task-based to Instruction-based Automated Log Analysis

LogSummary: Unstructured Log Summarization for Software Systems

HELP: Hierarchical Embeddings-based Log Parsing

Summarizing Unstructured Logs in Online Services

Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies

AWSOM-LP: An Effective Log Parsing Technique Using Pattern Recognition and Frequency Analysis

SMARTLOG: Place Error Log Statement by Deep Understanding of Log Intention

LogPrompt: Prompt Engineering Towards Zero-Shot and Interpretable Log Analysis

Analyzing Logs of Large-Scale Software Systems using Time Curves Visualization

AutoLog: A Log Sequence Synthesis Framework for Anomaly Detection

Logram: Efficient Log Parsing Using n-Gram Dictionaries

IPLog: An Efficient Log Parsing Method Based on Few-Shot Learning

LogAI: A Library for Log Analytics and Intelligence

LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models

Lemur: Log Parsing with Entropy Sampling and Chain-of-Thought Merging

Did We Miss Something Important? Studying and Exploring Variable-Aware Log Abstraction

A Survey on Automated Log Analysis for Reliability Engineering

FLAP: An End-to-End Event Log Analysis Platform for System Management

Prompting for Automatic Log Template Extraction