CourseGPT-zh: an Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimization

Zheyan Qu,Lu Yin,Zitong Yu,Wenbo Wang,Xing zhang

2024-05-08

Abstract:Large language models (LLMs) have demonstrated astonishing capabilities in natural language processing (NLP) tasks, sparking interest in their application to professional domains with higher specialized requirements. However, restricted access to closed-source LLMs via APIs and the difficulty in collecting massive high-quality datasets pose obstacles to the development of large language models in education fields of various courses. Given these challenges, we propose CourseGPT-zh, a course-oriented education LLM that supports customization and low-cost deployment. To address the comprehensiveness and diversity requirements of course-specific corpora, we design a high-quality question-answering corpus distillation framework incorporating prompt optimization, which effectively mines textbook knowledge and enhances its diversity. Moreover, considering the alignment of LLM responses with user needs, a novel method for discrete prompt optimization based on LLM-as-Judge is introduced. During optimization, this framework leverages the LLM's ability to reflect on and exploit error feedback and patterns, allowing for prompts that meet user needs and preferences while saving response length. Lastly, we obtain CourseGPT-zh based on the open-source LLM using parameter-efficient fine-tuning. Experimental results show that our discrete prompt optimization framework effectively improves the response quality of ChatGPT, and CourseGPT-zh exhibits strong professional capabilities in specialized knowledge question-answering, significantly outperforming comparable open-source models.

Computation and Language

What problem does this paper attempt to address?

This paper discusses how to build a course-oriented language model, CourseGPT-zh, for the education domain. Currently, although large language models perform well in natural language processing tasks, their application in the education field is limited due to the difficulty and high cost of acquiring high-quality specialized course datasets. To address these issues, the paper proposes a knowledge distillation-based framework, combined with prompt optimization, to generate high-quality question-answer pairs and customize the deployment of the education model at low cost.

The proposed methods in the paper include:
1. Designing a high-quality question-answering corpus distillation framework that effectively mines textbook knowledge and enhances its diversity through prompt optimization.
2. Introducing a novel discrete prompt optimization method based on LLM-as-Judge, which optimizes prompts to meet user needs and save response length by leveraging LLM's introspection and error feedback capabilities.
3. Using a parameter-efficient fine-tuning method to build CourseGPT-zh from an open-source LLM.

The experimental results demonstrate that the discrete prompt optimization framework effectively improves the response quality of ChatGPT, and CourseGPT-zh exhibits strong abilities in professional knowledge questioning, significantly outperforming other open-source models. This work aims to reduce barriers to using large language models in the education field and enhance the professionalism and efficiency of the models.