Panda LLM: Training Data and Evaluation for Open-Sourced Chinese Instruction-Following Large Language Models

Fangkai Jiao,Bosheng Ding,Tianze Luo,Zhanfeng Mo

DOI: https://doi.org/10.48550/arXiv.2305.03025

2023-05-05

Abstract:This project focuses on enhancing open-source large language models through instruction-tuning and providing comprehensive evaluations of their performance. We explore how various training data factors, such as quantity, quality, and linguistic distribution, influence the performance of instruction-tuned models trained on publicly accessible high-quality instruction datasets for both English and Chinese languages. Our goal is to supplement evaluation with quantitative analyses, providing valuable insights for the continued advancement of open-source chat models. Our model, data, and code are publicly available for others to use and build upon.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to improve the performance of open - source large - language models (LLMs) on instruction - following tasks, especially their performance in the Chinese context. Specifically, the paper focuses on the following aspects:

1. **Improving model performance**: Through the instruction - tuning technique, use a high - quality Chinese instruction dataset to fine - tune the base model, so as to improve the model's performance in various natural - language - processing tasks.

2. **Evaluating model performance**: Conduct a comprehensive performance evaluation of different open - source Chinese large - language models, including tests in reasoning ability, multi - round conversations, etc.

3. **Promoting the development of the open - source community**: Provide training data, model checkpoints, and source code to promote the development of the open - source community, enabling more researchers and developers to utilize these resources and promote the progress of large - language - model technology.

4. **Addressing the limitations of existing models**: Although existing commercial large - language models are powerful, they have deficiencies in the credibility of generated results, model transparency, and data security. By developing the open - source Panda LLM model, the paper aims to solve these problems and make the model more reliable, transparent, and customizable.

Through these efforts, the paper hopes to provide a high - performance, reliable, and transparent open - source solution for Chinese instruction - following tasks and promote further research and development in related fields.