Llama-Log: Optical Network Log Parsing with Fine-Tuned Large Language Model

L. Xiangbin,Yue Pang,Yanli Liu,Yahang Huan,Min Zhang,Danshi Wang

DOI: https://doi.org/10.1109/icocn63276.2024.10648375

2024-01-01

Abstract:This study proposes a method that involves finetuning the open-source large language model Llama2 for the task of parsing optical network logs. Through comparative experiments, we have validated the effectiveness of this method.

What problem does this paper attempt to address?

High-precision Online Log Parsing with Large Language Models

Xiaolei Chen,Jie Shi,Jia Chen,Wei Wang,Peng Wang

DOI: https://doi.org/10.1145/3639478.3643112

2024-04-14

Abstract:System logs are vital for diagnosing system failures, with log parsing converting unstructured logs into structured data. Existing methods fall into two categories: non-deep-learning approaches cluster logs based on stats but often miss semantic information, resulting in poor performance. Deep-learning approaches excel at identifying variables and constants but often lack generalizability beyond training data. And they always suffer from low efficiency. This paper proposes a novel LLM-based log parsing approach, named Hooglle, to address these challenges. Leveraging a large language model, Hooglle extracts templates for precise and generalized parsing. To overcome the efficiency issue, we propose a prefix-tree-based full-matching strategy which significantly improves parsing efficiency. Extensive evaluation across real-world datasets showcases Hooglle's superior performance on 16 public benchmark datasets.

Computer Science
LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models

Aoxiao Zhong,Dengyao Mo,Guiyang Liu,Jinbu Liu,Qingda Lu,Qi Zhou,Jiesheng Wu,Quanzheng Li,Qingsong Wen

2024-08-25

Abstract:Logs are ubiquitous digital footprints, playing an indispensable role in system diagnostics, security analysis, and performance optimization. The extraction of actionable insights from logs is critically dependent on the log parsing process, which converts raw logs into structured formats for downstream analysis. Yet, the complexities of contemporary systems and the dynamic nature of logs pose significant challenges to existing automatic parsing techniques. The emergence of Large Language Models (LLM) offers new horizons. With their expansive knowledge and contextual prowess, LLMs have been transformative across diverse applications. Building on this, we introduce LogParser-LLM, a novel log parser integrated with LLM capabilities. This union seamlessly blends semantic insights with statistical nuances, obviating the need for hyper-parameter tuning and labeled training data, while ensuring rapid adaptability through online parsing. Further deepening our exploration, we address the intricate challenge of parsing granularity, proposing a new metric and integrating human interactions to allow users to calibrate granularity to their specific needs. Our method's efficacy is empirically demonstrated through evaluations on the Loghub-2k and the large-scale LogPub benchmark. In evaluations on the LogPub benchmark, involving an average of 3.6 million logs per dataset across 14 datasets, our LogParser-LLM requires only 272.5 LLM invocations on average, achieving a 90.6% F1 score for grouping accuracy and an 81.1% for parsing accuracy. These results demonstrate the method's high efficiency and accuracy, outperforming current state-of-the-art log parsers, including pattern-based, neural network-based, and existing LLM-enhanced approaches.

Software Engineering,Artificial Intelligence
LLM-powered Zero-shot Online Log Parsing

Chen Zhi,Liye Cheng,Meilin Liu,Xinkui Zhao,Yueshen Xu,Shuiguang Deng

DOI: https://doi.org/10.1109/icws62655.2024.00106

2024-01-01

Abstract:Log parsing is an essential step for log analysis, which transforms raw log messages into structured form by extracting the log templates. Automatic log parsing have been the subject of extensive research. Recently, several studies have explored improving the performance of automatic log parsing via deep-learning-based approaches, especially for the pre-trained language models. However, the use of such large-scale language models for log parsing encounters several challenges, including hallucinations, high-cost and labelling efforts. To address these challenges, this paper introduces YALP, a zero-shot log parsing solution that tackles the aforementioned challenges by utilizing the capabilities of ChatGPT in conjunction with traditional methods, without incorporating user labelling. Our experiments on 16 public log datasets shows that our method outperforms several popular traditional methods in commonly used evaluation metrics. In comparison to directly utilizing GPT for log parsing tasks, our methods demonstrates significant improvements in both efficiency and effectiveness.
LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing

Zeyang Ma,An Ran Chen,Dong Jae Kim,Tse-Hsun Chen,Shaowei Wang

DOI: https://doi.org/10.1145/3597503.3639150

2024-04-28

Abstract:Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.

Software Engineering,Artificial Intelligence
LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models

Zeyang Ma,Dong Jae Kim,Tse-Hsun Chen

2024-11-02

Abstract:Log parsing is a critical step that transforms unstructured log data into structured formats, facilitating subsequent log-based analysis. Traditional syntax-based log parsers are efficient and effective, but they often experience decreased accuracy when processing logs that deviate from the predefined rules. Recently, large language models (LLM) based log parsers have shown superior parsing accuracy. However, existing LLM-based parsers face three main challenges: 1)time-consuming and labor-intensive manual labeling for fine-tuning or in-context learning, 2)increased parsing costs due to the vast volume of log data and limited context size of LLMs, and 3)privacy risks from using commercial models like ChatGPT with sensitive log information. To overcome these limitations, this paper introduces OpenLogParser, an unsupervised log parsing approach that leverages open-source LLMs (i.e., Llama3-8B) to enhance privacy and reduce operational costs while achieving state-of-the-art parsing accuracy. OpenLogParser first groups logs with similar static text but varying dynamic variables using a fixed-depth grouping tree. It then parses logs within these groups using three components: i)similarity scoring-based retrieval augmented generation: selects diverse logs within each group based on Jaccard similarity, helping the LLM distinguish between static text and dynamic variables; ii)self-reflection: iteratively query LLMs to refine log templates to improve parsing accuracy; and iii) log template memory: stores parsed templates to reduce LLM queries for improved parsing efficiency. Our evaluation on LogHub-2.0 shows that OpenLogParser achieves 25% higher parsing accuracy and processes logs 2.7 times faster compared to state-of-the-art LLM-based parsers. In short, OpenLogParser addresses privacy and cost concerns of using commercial LLMs while achieving state-of-the-arts parsing efficiency and accuracy.

Software Engineering,Artificial Intelligence
Adapting Large Language Models to Log Analysis with Interpretable Domain Knowledge

Yuhe Ji,Yilun Liu,Feiyu Yao,Minggui He,Shimin Tao,Xiaofeng Zhao,Su Chang,Xinhua Yang,Weibin Meng,Yuming Xie,Boxing Chen,Hao Yang

2024-12-02

Abstract:The increasing complexity of computer systems necessitates innovative approaches to fault and error management, going beyond traditional manual log analysis. While existing solutions using large language models (LLMs) show promise, they are limited by a gap between natural and domain-specific languages, which restricts their effectiveness in real-world applications. Our approach addresses these limitations by integrating interpretable domain knowledge into open-source LLMs through continual pre-training (CPT), enhancing performance on log tasks while retaining natural language processing capabilities. We created a comprehensive dataset, NLPLog, with over 250,000 question-answer pairs to facilitate this integration. Our model, SuperLog, trained with this dataset, achieves the best performance across four log analysis tasks, surpassing the second-best model by an average of 12.01%. Our contributions include a novel CPT paradigm that significantly improves model performance, the development of SuperLog with state-of-the-art results, and the release of a large-scale dataset to support further research in this domain.

Computation and Language,Software Engineering
Self-Evolutionary Group-wise Log Parsing Based on Large Language Model

Changhua Pei,Zihan Liu,Jianhui Li,Erhan Zhang,Le Zhang,Haiming Zhang,Wei Chen,Dan Pei,Gaogang Xie

DOI: https://doi.org/10.1109/issre62328.2024.00016

2024-01-01

Abstract:Log parsing involves extracting appropriate templates from semi-structured logs, providing foundational information for downstream log analysis tasks such as anomaly detection and log comprehension. Initially, the task of log parsing was approached by domain experts who manually designed heuristic rules to extract templates. However, the effectiveness of these manual rules deteriorates when certain characteristics of a new log dataset do not conform to the pre-designed rules. To address these issues, introducing large language models (LLM) into log parsing has yielded promising results. Nevertheless, there are two limitations: one is the reliance on manually annotated templates within the prompt, and the other is the low efficiency of log processing. To address these challenges, we propose a self-evolving method called SelfLog, which, on the one hand, uses similar pairs extracted by LLM itself in the historical data to act as the prompt of a new log, allowing the model to learn in a self-evolution and labeling-free way. On the other hand, we propose an N-Gram-based grouper and log hitter. This approach not only improves the parsing performance of LLM by extracting the templates in a group-wise way instead of a log-wise way but also significantly reduces the unnecessary calling to LLMs for those logs whose group template is already extracted in history. We evaluate the performance and efficiency of SelfLog on 16 public datasets, involving tens of millions of logs, and the experiments demonstrate that SelfLog has achieved state-of-the-art (SOTA) levels in 0.975’s GA, and 0.942’s PA. More importantly, without sacrificing accuracy, the processing speed has reached a remarkable 45,000 logs per second.
When Large Language Models Meet Optical Networks: Paving the Way for Automation

Danshi Wang,Yidi Wang,Xiaotian Jiang,Yao Zhang,Yue Pang,Min Zhang

2024-06-25

Abstract:Since the advent of GPT, large language models (LLMs) have brought about revolutionary advancements in all walks of life. As a superior natural language processing (NLP) technology, LLMs have consistently achieved state-of-the-art performance on numerous areas. However, LLMs are considered to be general-purpose models for NLP tasks, which may encounter challenges when applied to complex tasks in specialized fields such as optical networks. In this study, we propose a framework of LLM-empowered optical networks, facilitating intelligent control of the physical layer and efficient interaction with the application layer through an LLM-driven agent (AI-Agent) deployed in the control layer. The AI-Agent can leverage external tools and extract domain knowledge from a comprehensive resource library specifically established for optical networks. This is achieved through user input and well-crafted prompts, enabling the generation of control instructions and result representations for autonomous operation and maintenance in optical networks. To improve LLM's capability in professional fields and stimulate its potential on complex tasks, the details of performing prompt engineering, establishing domain knowledge library, and implementing complex tasks are illustrated in this study. Moreover, the proposed framework is verified on two typical tasks: network alarm analysis and network performance optimization. The good response accuracies and sematic similarities of 2,400 test situations exhibit the great potential of LLM in optical networks.

Networking and Internet Architecture,Artificial Intelligence,Computation and Language,Systems and Control
LUNAR: Unsupervised LLM-based Log Parsing

Junjie Huang,Zhihan Jiang,Zhuangbin Chen,Michael R. Lyu

2024-08-09

Abstract:Log parsing serves as an essential prerequisite for various log analysis tasks. Recent advancements in this field have improved parsing accuracy by leveraging the semantics in logs through fine-tuning large language models (LLMs) or learning from in-context demonstrations. However, these methods heavily depend on labeled examples to achieve optimal performance. In practice, collecting sufficient labeled data is challenging due to the large scale and continuous evolution of logs, leading to performance degradation of existing log parsers after deployment. To address this issue, we propose LUNAR, an unsupervised LLM-based method for efficient and off-the-shelf log parsing. Our key insight is that while LLMs may struggle with direct log parsing, their performance can be significantly enhanced through comparative analysis across multiple logs that differ only in their parameter parts. We refer to such groups of logs as Log Contrastive Units (LCUs). Given the vast volume of logs, obtaining LCUs is difficult. Therefore, LUNAR introduces a hybrid ranking scheme to effectively search for LCUs by jointly considering the commonality and variability among logs. Additionally, LUNAR crafts a novel parsing prompt for LLMs to identify contrastive patterns and extract meaningful log structures from LCUs. Experiments on large-scale public datasets demonstrate that LUNAR significantly outperforms state-of-the-art log parsers in terms of accuracy and efficiency, providing an effective and scalable solution for real-world deployment. \footnote{The code and data are available at \url{<a class="link-external link-https" href="https://github.com/Jun-jie-Huang/LUNAR" rel="external noopener nofollow">this https URL</a>}}.

Software Engineering
LogLLM: Log-based Anomaly Detection Using Large Language Models

Wei Guan,Jian Cao,Shiyou Qian,Jianqi Gao

2024-11-13

Abstract:Software systems often record important runtime information in logs to help with troubleshooting. Log-based anomaly detection has become a key research area that aims to identify system issues through log data, ultimately enhancing the reliability of software systems. Traditional deep learning methods often struggle to capture the semantic information embedded in log data, which is typically organized in natural language. In this paper, we propose LogLLM, a log-based anomaly detection framework that leverages large language models (LLMs). LogLLM employs BERT for extracting semantic vectors from log messages, while utilizing Llama, a transformer decoder-based model, for classifying log sequences. Additionally, we introduce a projector to align the vector representation spaces of BERT and Llama, ensuring a cohesive understanding of log semantics. Unlike conventional methods that require log parsers to extract templates, LogLLM preprocesses log messages with regular expressions, streamlining the entire process. Our framework is trained through a novel three-stage procedure designed to enhance performance and adaptability. Experimental results across four public datasets demonstrate that LogLLM outperforms state-of-the-art methods. Even when handling unstable logs, it effectively captures the semantic meaning of log messages and detects anomalies accurately.

Software Engineering,Artificial Intelligence
Log Parsing with Self-Generated In-Context Learning and Self-Correction

Yifan Wu,Siyu Yu,Ying Li

2024-06-05

Abstract:Log parsing transforms log messages into structured formats, serving as a crucial step for log analysis. Despite a variety of log parsing methods that have been proposed, their performance on evolving log data remains unsatisfactory due to reliance on human-crafted rules or learning-based models with limited training data. The recent emergence of large language models (LLMs) has demonstrated strong abilities in understanding natural language and code, making it promising to apply LLMs for log parsing. Consequently, several studies have proposed LLM-based log parsers. However, LLMs may produce inaccurate templates, and existing LLM-based log parsers directly use the template generated by the LLM as the parsing result, hindering the accuracy of log parsing. Furthermore, these log parsers depend heavily on historical log data as demonstrations, which poses challenges in maintaining accuracy when dealing with scarce historical log data or evolving log data. To address these challenges, we propose AdaParser, an effective and adaptive log parsing framework using LLMs with self-generated in-context learning (SG-ICL) and self-correction. To facilitate accurate log parsing, AdaParser incorporates a novel component, a template corrector, which utilizes the LLM to correct potential parsing errors in the templates it generates. In addition, AdaParser maintains a dynamic candidate set composed of previously generated templates as demonstrations to adapt evolving log data. Extensive experiments on public large-scale datasets show that AdaParser outperforms state-of-the-art methods across all metrics, even in zero-shot scenarios. Moreover, when integrated with different LLMs, AdaParser consistently enhances the performance of the utilized LLMs by a large margin.

Software Engineering
LILAC: Log Parsing using LLMs with Adaptive Parsing Cache

Zhihan Jiang,Jinyang Liu,Zhuangbin Chen,Yichen Li,Junjie Huang,Yintong Huo,Pinjia He,Jiazhen Gu,Michael R. Lyu

DOI: https://doi.org/10.1145/3643733

2024-07-12

Abstract:Log parsing transforms log messages into structured formats, serving as the prerequisite step for various log analysis tasks. Although a variety of log parsing approaches have been proposed, their performance on complicated log data remains compromised due to the use of human-crafted rules or learning-based models with limited training data. The recent emergence of powerful large language models (LLMs) demonstrates their vast pre-trained knowledge related to code and logging, making it promising to apply LLMs for log parsing. However, their lack of specialized log parsing capabilities currently hinders their parsing accuracy. Moreover, the inherent inconsistent answers, as well as the substantial overhead, prevent the practical adoption of LLM-based log parsing. To address these challenges, we propose LILAC, the first practical Log parsIng framework using LLMs with Adaptive parsing Cache. To facilitate accurate and robust log parsing, LILAC leverages the in-context learning (ICL) capability of the LLM by performing a hierarchical candidate sampling algorithm and selecting high-quality demonstrations. Furthermore, LILAC incorporates a novel component, an adaptive parsing cache, to store and refine the templates generated by the LLM. It helps mitigate LLM's inefficiency issue by enabling rapid retrieval of previously processed log templates. In this process, LILAC adaptively updates the templates within the parsing cache to ensure the consistency of parsed results. The extensive evaluation on public large-scale datasets shows that LILAC outperforms state-of-the-art methods by 69.5% in terms of the average F1 score of template accuracy. In addition, LILAC reduces the query times to LLMs by several orders of magnitude, achieving a comparable efficiency to the fastest baseline.
DLLog: An Online Log Parsing Approach for Large-Scale System

Hailong Cheng,Shi Ying,Xiaoyu Duan,Wanli Yuan

DOI: https://doi.org/10.1155/2024/5961993

IF: 8.993

2024-04-16

International Journal of Intelligent Systems

Abstract:Syslog is a critical data source for analyzing system problems. Converting unstructured log entries into structured log data is necessary for effective log analysis. However, existing log parsing methods demonstrate promising accuracy on limited datasets, but their generalizability and precision are uncertain when applied to diverse log data. Enhancements in these areas are necessary. This paper proposes an online log parsing method called DLLog, which is based on deep learning and has the longest common subsequence. DLLog utilizes the GRU neural network to mine template words and applies the longest common subsequence to parse log entries in real-time. In the offline stage, DLLog combines multiple log features to accurately extract the template words, creating a log template set to assist online log parsing. In the online stage, DLLog parses log entries by calculating the matching degree between the real-time log entry and the log template in the log template set. This method also supports the incremental update of the log template set to handle new log entries generated by systems. We summarized the previous works and validated DLLog using real log data collected from 16 systems. The results demonstrate that DLLog achieves high parsing accuracy, universality, and adaptability.

computer science, artificial intelligence
Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies

Yilun Liu,Shimin Tao,Weibin Meng,Jingyu Wang,Wenbing Ma,Yanqing Zhao,Yuhang Chen,Hao Yang,Yanfei Jiang,Xun Chen

DOI: https://doi.org/10.48550/arXiv.2308.07610

2024-01-26

Abstract:Automated log analysis is crucial in modern software-intensive systems for facilitating program comprehension throughout software maintenance and engineering life cycles. Existing methods perform tasks such as log parsing and log anomaly detection by providing a single prediction value without interpretation. However, given the increasing volume of system events, the limited interpretability of analysis results hinders analysts' comprehension of program status and their ability to take appropriate actions. Moreover, these methods require substantial in-domain training data, and their performance declines sharply (by up to 62.5%) in online scenarios involving unseen logs from new domains, a common occurrence due to rapid software updates. In this paper, we propose LogPrompt, a novel interpretable log analysis approach for online scenarios. LogPrompt employs large language models (LLMs) to perform online log analysis tasks via a suite of advanced prompt strategies tailored for log tasks, which enhances LLMs' performance by up to 380.7% compared with simple prompts. Experiments on nine publicly available evaluation datasets across two tasks demonstrate that LogPrompt, despite requiring no in-domain training, outperforms existing approaches trained on thousands of logs by up to 55.9%. We also conduct a human evaluation of LogPrompt's interpretability, with six practitioners possessing over 10 years of experience, who highly rated the generated content in terms of usefulness and readability (averagely 4.42/5). LogPrompt also exhibits remarkable compatibility with open-source and smaller-scale LLMs, making it flexible for practical deployment. Code of LogPrompt is available at <a class="link-external link-https" href="https://github.com/lunyiliu/LogPrompt" rel="external noopener nofollow">this https URL</a>.

Software Engineering,Computation and Language
YuLan: An Open-source Large Language Model

Yutao Zhu,Kun Zhou,Kelong Mao,Wentong Chen,Yiding Sun,Zhipeng Chen,Qian Cao,Yihan Wu,Yushuo Chen,Feng Wang,Lei Zhang,Junyi Li,Xiaolei Wang,Lei Wang,Beichen Zhang,Zican Dong,Xiaoxue Cheng,Yuhan Chen,Xinyu Tang,Yupeng Hou,Qiangqiang Ren,Xincheng Pang,Shufang Xie,Wayne Xin Zhao,Zhicheng Dou,Jiaxin Mao,Yankai Lin,Ruihua Song,Jun Xu,Xu Chen,Rui Yan,Zhewei Wei,Di Hu,Wenbing Huang,Ze-Feng Gao,Yueguo Chen,Weizheng Lu,Ji-Rong Wen

2024-06-28

Abstract:Large language models (LLMs) have become the foundation of many applications, leveraging their extensive capabilities in processing and understanding natural language. While many open-source LLMs have been released with technical reports, the lack of training details hinders further research and development. This paper presents the development of YuLan, a series of open-source LLMs with $12$ billion parameters. The base model of YuLan is pre-trained on approximately $1.7$T tokens derived from a diverse corpus, including massive English, Chinese, and multilingual texts. We design a three-stage pre-training method to enhance YuLan's overall capabilities. Subsequent phases of training incorporate instruction-tuning and human alignment, employing a substantial volume of high-quality synthesized data. To facilitate the learning of complex and long-tail knowledge, we devise a curriculum-learning framework throughout across these stages, which helps LLMs learn knowledge in an easy-to-hard manner. YuLan's training is finished on Jan, 2024 and has achieved performance on par with state-of-the-art LLMs across various English and Chinese benchmarks. This paper outlines a comprehensive technical roadmap for developing LLMs from scratch. Our model and codes are available at <a class="link-external link-https" href="https://github.com/RUC-GSAI/YuLan-Chat" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence
LogEval: A Comprehensive Benchmark Suite for Large Language Models In Log Analysis

Tianyu Cui,Shiyu Ma,Ziang Chen,Tong Xiao,Shimin Tao,Yilun Liu,Shenglin Zhang,Duoming Lin,Changchang Liu,Yuzhe Cai,Weibin Meng,Yongqian Sun,Dan Pei

2024-07-02

Abstract:Log analysis is crucial for ensuring the orderly and stable operation of information systems, particularly in the field of Artificial Intelligence for IT Operations (AIOps). Large Language Models (LLMs) have demonstrated significant potential in natural language processing tasks. In the AIOps domain, they excel in tasks such as anomaly detection, root cause analysis of faults, operations and maintenance script generation, and alert information summarization. However, the performance of current LLMs in log analysis tasks remains inadequately validated. To address this gap, we introduce LogEval, a comprehensive benchmark suite designed to evaluate the capabilities of LLMs in various log analysis tasks for the first time. This benchmark covers tasks such as log parsing, log anomaly detection, log fault diagnosis, and log summarization. LogEval evaluates each task using 4,000 publicly available log data entries and employs 15 different prompts for each task to ensure a thorough and fair assessment. By rigorously evaluating leading LLMs, we demonstrate the impact of various LLM technologies on log analysis performance, focusing on aspects such as self-consistency and few-shot contextual learning. We also discuss findings related to model quantification, Chinese-English question-answering evaluation, and prompt engineering. These findings provide insights into the strengths and weaknesses of LLMs in multilingual environments and the effectiveness of different prompt strategies. Various evaluation methods are employed for different tasks to accurately measure the performance of LLMs in log analysis, ensuring a comprehensive assessment. The insights gained from LogEvals evaluation reveal the strengths and limitations of LLMs in log analysis tasks, providing valuable guidance for researchers and practitioners.

Computation and Language,Information Retrieval
Natural Language Processing-based Model for Log Anomaly Detection

Zezhou Li,Jing Zhang,Xianbo Zhang,Feng Lin,Chao Wang,Xingye Cai

DOI: https://doi.org/10.1109/seai55746.2022.9832400

2022-01-01

Abstract:Logs are widely used in IT industry and the anomaly detection of logs is essential to identify the running status of systems. Conventional methods solving this problem require sophisticated rule-based regulations and intensive labor input. In this paper, we propose a new model based on natural language processing techniques. In order to modify the feature extraction and to improve the vector quality of log templates, Part-of-Speech (PoS) and Named Entity Recognition (NER) are employed in our model, which leads to the less involvement of regulation-based rule and a modification of the template vector thanks to the weight vector by NER. The PoS property of each token in the template is firstly analyzed, which also reduces labor involvement and helps for better weight allocation. The weight investigation on tokens of the template is introduced to modify the template vector. And the final detection based on the modified vector of templates is realized by deep neural networks (DNNs). The effectiveness of our model is tested on three datasets, and compared with two state-of-the-art models. The evaluation results prove that our model achieves better log anomaly detection.
A Comparative Study on Large Language Models for Log Parsing

Merve Astekin,Max Hort,Leon Moonen

DOI: https://doi.org/10.1145/3674805.3686684

2024-09-04

Abstract:Background: Log messages provide valuable information about the status of software systems. This information is provided in an unstructured fashion and automated approaches are applied to extract relevant parameters. To ease this process, log parsing can be applied, which transforms log messages into structured log templates. Recent advances in language models have led to several studies that apply ChatGPT to the task of log parsing with promising results. However, the performance of other state-of-the-art large language models (LLMs) on the log parsing task remains unclear. Aims: In this study, we investigate the current capability of state-of-the-art LLMs to perform log parsing. Method: We select six recent LLMs, including both paid proprietary (GPT-3.5, Claude 2.1) and four free-to-use open models, and compare their performance on system logs obtained from a selection of mature open-source projects. We design two different prompting approaches and apply the LLMs on 1, 354 log templates across 16 different projects. We evaluate their effectiveness, in the number of correctly identified templates, and the syntactic similarity between the generated templates and the ground truth. Results: We found that free-to-use models are able to compete with paid models, with CodeLlama extracting 10% more log templates correctly than GPT-3.5. Moreover, we provide qualitative insights into the usability of language models (e.g., how easy it is to use their responses). Conclusions: Our results reveal that some of the smaller, free-to-use LLMs can considerably assist log parsing compared to their paid proprietary competitors, especially code-specialized models.

Software Engineering,Computation and Language
Demonstration-Free: Towards More Practical Log Parsing with Large Language Models

Yi Xiao,Van-Hoang Le,Hongyu Zhang

DOI: https://doi.org/10.1145/3691620.3694994

2024-01-01

Abstract:Log parsing, the process of converting raw log messages into structured formats, is an important initial step for automated analysis of logs of large-scale software systems. Traditional log parsers often rely on heuristics or handcrafted features, which may not generalize well across diverse log sources or require extensive model tuning. Recently, some log parsers have utilized powerful generative capabilities of large language models (LLMs). However, they heavily rely on demonstration examples, resulting in substantial overhead in LLM invocations. To address these issues, we propose LogBatcher, a cost-effective LLM-based log parser that requires no training process or labeled data. To leverage latent characteristics of log data and reduce the overhead, we divide logs into several partitions through clustering. Then we perform a cache matching process to match logs with previously parsed log templates. Finally, we provide LLMs with better prompt context specialized for log parsing by batching a group of logs from each partition. We have conducted experiments on 16 public log datasets and the results show that LogBatcher is effective and efficient for log parsing.

Llama-Log: Optical Network Log Parsing with Fine-Tuned Large Language Model

High-precision Online Log Parsing with Large Language Models

LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models

LLM-powered Zero-shot Online Log Parsing

LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing

LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models

Adapting Large Language Models to Log Analysis with Interpretable Domain Knowledge

Self-Evolutionary Group-wise Log Parsing Based on Large Language Model

When Large Language Models Meet Optical Networks: Paving the Way for Automation

LUNAR: Unsupervised LLM-based Log Parsing

LogLLM: Log-based Anomaly Detection Using Large Language Models

Log Parsing with Self-Generated In-Context Learning and Self-Correction

LILAC: Log Parsing using LLMs with Adaptive Parsing Cache

DLLog: An Online Log Parsing Approach for Large-Scale System

Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies

YuLan: An Open-source Large Language Model

LogEval: A Comprehensive Benchmark Suite for Large Language Models In Log Analysis

Natural Language Processing-based Model for Log Anomaly Detection

A Comparative Study on Large Language Models for Log Parsing

Demonstration-Free: Towards More Practical Log Parsing with Large Language Models