Knowledge Tagging with Large Language Model based Multi-Agent System

Hang Li,Tianlong Xu,Ethan Chang,Qingsong Wen

2024-09-13

Abstract:Knowledge tagging for questions is vital in modern intelligent educational applications, including learning progress diagnosis, practice question recommendations, and course content organization. Traditionally, these annotations have been performed by pedagogical experts, as the task demands not only a deep semantic understanding of question stems and knowledge definitions but also a strong ability to link problem-solving logic with relevant knowledge concepts. With the advent of advanced natural language processing (NLP) algorithms, such as pre-trained language models and large language models (LLMs), pioneering studies have explored automating the knowledge tagging process using various machine learning models. In this paper, we investigate the use of a multi-agent system to address the limitations of previous algorithms, particularly in handling complex cases involving intricate knowledge definitions and strict numerical constraints. By demonstrating its superior performance on the publicly available math question knowledge tagging dataset, MathKnowCT, we highlight the significant potential of an LLM-based multi-agent system in overcoming the challenges that previous methods have encountered. Finally, through an in-depth discussion of the implications of automating knowledge tagging, we underscore the promising results of deploying LLM-based algorithms in educational contexts.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to automatically perform knowledge annotation in modern intelligent education applications. Specifically, knowledge annotation is crucial for learning progress diagnosis, exercise recommendation, and course content organization. Traditionally, these annotation tasks are completed manually by education experts, which requires a deep understanding of the question text and knowledge definitions, and the ability to connect the problem - solving logic with relevant knowledge concepts. However, with the rapid increase of online content, the manual method can no longer meet the annotation needs of a large amount of online question data, so there is an urgent need for automated methods to improve efficiency and accuracy.

The paper proposes a method based on multi - agent systems (MAS), taking advantage of the capabilities of large - language models (LLMs). By decomposing complex tasks into multiple independent subtasks, it simplifies the entire annotation process and improves the reliability of judgment generation. This method is particularly suitable for handling situations involving complex knowledge definitions and strict numerical constraints. Through experiments on the publicly available dataset MathKnowCT, the paper shows that the proposed multi - agent system has significant potential in overcoming the challenges encountered by previous methods.