LUK: Empowering Log Understanding with Expert Knowledge from Large Language Models

Lipeng Ma,Weidong Yang,Sihang Jiang,Ben Fei,Mingjie Zhou,Shuhao Li,Bo Xu,Yanghua Xiao

2024-09-03

Abstract:Logs play a critical role in providing essential information for system monitoring and troubleshooting. Recently, with the success of pre-trained language models (PLMs) and large language models (LLMs) in natural language processing (NLP), smaller PLMs (such as BERT) and LLMs (like ChatGPT) have become the current mainstream approaches for log analysis. While LLMs possess rich knowledge, their high computational costs and unstable performance make LLMs impractical for analyzing logs directly. In contrast, smaller PLMs can be fine-tuned for specific tasks even with limited computational resources, making them more practical. However, these smaller PLMs face challenges in understanding logs comprehensively due to their limited expert knowledge. To better utilize the knowledge embedded within LLMs for log understanding, this paper introduces a novel knowledge enhancement framework, called LUK, which acquires expert knowledge from LLMs to empower log understanding on a smaller PLM. Specifically, we design a multi-expert collaboration framework based on LLMs consisting of different roles to acquire expert knowledge. In addition, we propose two novel pre-training tasks to enhance the log pre-training with expert knowledge. LUK achieves state-of-the-art results on different log analysis tasks and extensive experiments demonstrate expert knowledge from LLMs can be utilized more effectively to understand logs.

Software Engineering,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address key issues in log understanding, particularly how to leverage expert knowledge from large-scale language models (LLMs) to enhance the log understanding capabilities of smaller pre-trained language models (PLMs). Specifically, the paper proposes a new framework called LUK, which addresses this issue in the following ways:

1. **Effectively Acquiring Expert Knowledge**: Although large-scale language models possess rich knowledge, they face high computational costs and performance instability when directly used for log analysis. Therefore, LUK designs a multi-expert collaboration framework to extract expert knowledge from LLMs. This framework is based on role division, including "Director," "Executor," and "Evaluator," and ensures that each role can efficiently collaborate through role-playing to generate high-quality expert knowledge.

2. **Knowledge-Enhanced Pre-training**: To integrate this expert knowledge into smaller PLMs, the paper proposes two novel pre-training tasks—token-level prediction and sentence-level semantic alignment. These tasks help embed background knowledge into the pre-training process, thereby improving log understanding capabilities.

3. **Fine-tuning for Downstream Tasks**: Finally, the knowledge-enhanced PLMs can be fine-tuned for different downstream log analysis tasks, such as anomaly detection and fault identification, to achieve better performance.

The paper validates the effectiveness of LUK in various log analysis tasks through experiments and demonstrates its significant advantages in resource-constrained scenarios. Overall, LUK provides an effective method to utilize expert knowledge from large-scale language models, thereby improving the log understanding capabilities of smaller pre-trained models.