Abstract:Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.

Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning

YUAN 2.0: A Large Language Model with Localized Filtering-based Attention

LanYUAN, a GPT Large Model Using Curriculum Learning and Sparse Attention.

YuLan: An Open-source Large Language Model

YuLan-Mini: An Open Data-efficient Language Model

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Baichuan 2: Open Large-scale Language Models

PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

A Survey of Large Language Models

Qwen Technical Report

BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

CPM: A large-scale generative Chinese Pre-trained language model

Zero and R2D2: A Large-scale Chinese Cross-modal Benchmark and A Vision-Language Framework

Research on the Application and Optimization Strategies of Deep Learning in Large Language Models

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Models

Multilevel Large Language Models for Everyone