LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing

Zeyang Ma,An Ran Chen,Dong Jae Kim,Tse-Hsun Chen,Shaowei Wang

DOI: https://doi.org/10.1145/3597503.3639150

2024-04-28

Abstract:Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.

Software Engineering,Artificial Intelligence

What problem does this paper attempt to address?

This paper discusses the problem of log parsing using Large Language Models (LLMs). Traditional log parsers face challenges in accuracy due to the diversity of log formats, which directly impacts the performance of downstream log analysis tasks. The study proposes a log parser called LLMParser, which is based on LLMs and fine-tuned with a small number of examples. By evaluating log data from 16 open source systems, the results show that LLMParser surpasses state-of-the-art parsers in terms of average parsing accuracy (96%). The study also analyzes the impact of training size, model size, and pre-training LLM on parsing accuracy, finding that smaller LLMs can be more effective than more complex ones, and pre-training does not always improve parsing accuracy. This research provides empirical evidence for the application of LLMs in log parsing and indicates directions and limitations for future research.