MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications

Yizhe Yang,Huashan Sun,Jiawei Li,Runheng Liu,Yinghao Li,Yuhang Liu,Heyan Huang,Yang Gao
DOI: https://doi.org/10.48550/arXiv.2310.15777
2023-10-29
Abstract:Large Language Models (LLMs) have demonstrated remarkable performance across various natural language tasks, marking significant strides towards general artificial intelligence. While general artificial intelligence is leveraged by developing increasingly large-scale models, there could be another branch to develop lightweight custom models that better serve certain domains, taking into account the high cost of training and deploying LLMs and the scarcity of resources. In this paper, we present MindLLM, a novel series of bilingual lightweight large language models, trained from scratch, alleviating such burdens by offering models with 1.3 billion and 3 billion parameters. A thorough account of experiences accrued during large model development is given, covering every step of the process, including data construction, model architecture, evaluation, and applications. Such insights are hopefully valuable for fellow academics and developers. MindLLM consistently matches or surpasses the performance of other open-source larger models on some public benchmarks. We also introduce an innovative instruction tuning framework tailored for smaller models to enhance their capabilities efficiently. Moreover, we explore the application of MindLLM in specific vertical domains such as law and finance, underscoring the agility and adaptability of our lightweight models.
Computation and Language,Artificial Intelligence
What problem does this paper attempt to address?
### What problems does this paper attempt to solve? This paper aims to solve the problems of resource efficiency and adaptability in the application of large - language models (LLMs) in specific fields. Specifically, the author proposes a series of lightweight bilingual large - language models named MindLLM to address the following issues: 1. **Resource Consumption and Cost**: - Although large - language models perform well in various natural language tasks, their training and deployment costs are high, especially in the case of limited computing resources. - Lightweight models can reduce these costs, enabling more developers and researchers to use high - performance language models. 2. **Adaptability in Specific Fields**: - Many applications in specific fields do not require extensive general knowledge or complex multi - task processing capabilities, but rather need optimization for specific tasks and domain knowledge. - Through customized training and optimization, MindLLM can achieve excellent performance in specific fields such as law and finance while maintaining low computational requirements. 3. **The Relationship between Model Performance and Scale**: - Traditionally, increasing the number of model parameters is considered an effective way to improve performance, but this method is not always feasible in the case of limited resources. - Research shows that lightweight models can outperform larger - scale models in some tasks through carefully designed training strategies and data selection. 4. **The Effectiveness of Instruction Tuning**: - For lightweight models, how to enhance specific capabilities through instruction tuning is a challenge. - The paper proposes an entropy - based quality filtering strategy for constructing a high - quality instruction - tuning data set, thereby significantly improving the specific capabilities of lightweight models. 5. **The Stability of Cross - Language Learning**: - In a bilingual or multilingual environment, how to ensure the stability and consistency of the model between different languages is also an important issue. - MindLLM improves the performance of cross - language tasks by training bilingual data from scratch, avoiding the catastrophic forgetting problem that may be caused by transfer learning. In summary, through the development and evaluation of MindLLM, this paper aims to explore the potential of lightweight language models in terms of resource efficiency, adaptability in specific fields, and performance optimization.