Abstract:Large Language Models (LLMs), such as ChatGPT and GPT-4, have dramatically transformed natural language processing research and shown promising strides towards Artificial General Intelligence (AGI). Nonetheless, the high costs associated with training and deploying LLMs present substantial obstacles to transparent, accessible academic research. While several large language models, such as LLaMA, have been open-sourced by the community, these predominantly focus on English corpora, limiting their usefulness for other languages. In this paper, we propose a method to augment LLaMA with capabilities for understanding and generating Chinese text and its ability to follow instructions. We achieve this by extending LLaMA's existing vocabulary with an additional 20,000 Chinese tokens, thereby improving its encoding efficiency and semantic understanding of Chinese. We further incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets, significantly enhancing the model's ability to comprehend and execute instructions. Our experimental results indicate that the newly proposed model markedly enhances the original LLaMA's proficiency in understanding and generating Chinese content. Additionally, the results on the C-Eval dataset yield competitive performance among the models with several times the size of ours. We have made our pre-trained models, training scripts, and other resources available through GitHub, fostering open research for our community. Chinese LLaMA series: \url{<a class="link-external link-https" href="https://github.com/ymcui/Chinese-LLaMA-Alpaca" rel="external noopener nofollow">this https URL</a>} and Chinese Llama-2 series: \url{<a class="link-external link-https" href="https://github.com/ymcui/Chinese-LLaMA-Alpaca-2" rel="external noopener nofollow">this https URL</a>}

Sub-Character Tokenization for Chinese Pretrained Language Models

SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining

Character-Level Syntax Infusion in Pre-Trained Models for Chinese Semantic Role Labeling.

CharBERT: Character-aware Pre-trained Language Model

Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models

Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models

Learning Mutually Informed Representations for Characters and Subwords

ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

SubCharacter Chinese-English Neural Machine Translation with Wubi encoding

When is Char Better Than Subword: A Systematic Study of Segmentation Algorithms for Neural Machine Translation

Empirical Study on Character Level Neural Network Classifier for Chinese Text.

A Vocabulary-Free Multilingual Neural Tokenizer for End-to-End Task Learning

Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca

Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts

Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

N-LTP: A Open-source Neural Chinese Language Technology Platform with Pretrained Models.

2kenize: Tying Subword Sequences for Chinese Script Conversion

Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Tokenization Falling Short: On Subword Robustness in Large Language Models

Unsupervised Morphological Tree Tokenizer