TongGu: Mastering Classical Chinese Understanding with Knowledge-Grounded Large Language Models

Jiahuan Cao,Dezhi Peng,Peirong Zhang,Yongxin Shi,Yang Liu,Kai Ding,Lianwen Jin

2024-09-30

Abstract:Classical Chinese is a gateway to the rich heritage and wisdom of ancient China, yet its complexities pose formidable comprehension barriers for most modern people without specialized knowledge. While Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), they struggle with Classical Chinese Understanding (CCU), especially in data-demanding and knowledge-intensive tasks. In response to this dilemma, we propose \textbf{TongGu} (mean understanding ancient and modern), the first CCU-specific LLM, underpinned by three core contributions. First, we construct a two-stage instruction-tuning dataset ACCN-INS derived from rich classical Chinese corpora, aiming to unlock the full CCU potential of LLMs. Second, we propose Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting, enabling TongGu to acquire new capabilities while preserving its foundational knowledge. Third, we present a CCU Retrieval-Augmented Generation (CCU-RAG) technique to reduce hallucinations based on knowledge-grounding. Extensive experiments across 24 diverse CCU tasks validate TongGu's superior ability, underscoring the effectiveness of RAT and CCU-RAG. The model and dataset are available at \url{<a class="link-external link-https" href="https://github.com/SCUT-DLVCLab/TongGu-LLM" rel="external noopener nofollow">this https URL</a>}.

Computation and Language

What problem does this paper attempt to address?

The paper attempts to address the difficulties modern people encounter in understanding and processing Classical Chinese. Despite the excellent performance of large language models (LLMs) in natural language processing (NLP), they have limitations in understanding and processing Classical Chinese (Classical Chinese Understanding, CCU), especially in tasks that are data-intensive and knowledge-intensive. The paper proposes a large language model specifically designed for Classical Chinese understanding—TongGu, aiming to overcome these challenges. Specifically, the main contributions of the paper include:

1. **Constructing a specialized instruction-tuning dataset**: A two-stage instruction-tuning dataset, ACCN-INS, is proposed, derived from a rich corpus of Classical Chinese, aiming to unlock the full potential of LLMs in understanding Classical Chinese.

2. **Preventing catastrophic forgetting**: A Redundancy-Aware Tuning (RAT) method is proposed, which prevents catastrophic forgetting by identifying and freezing the most critical layers in the current task, allowing TongGu to acquire new capabilities while retaining foundational knowledge.

3. **Reducing hallucinations**: A knowledge retrieval-based generation technique (CCU-RAG) is introduced to reduce hallucinations in knowledge-intensive tasks, further enhancing TongGu's performance.

Through these methods, TongGu has demonstrated superior capabilities in 24 different Classical Chinese understanding tasks, validating the effectiveness of RAT and CCU-RAG.