Abstract:Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.

Scaling Laws for Discriminative Classification in Large Language Models

Scaling Laws for Multilingual Language Models

Temporal Scaling Law for Large Language Models

Evaluating and Mitigating Linguistic Discrimination in Large Language Models

Supervised Knowledge Makes Large Language Models Better In-context Learners

Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology

Performance Law of Large Language Models

Scaling Law for Language Models Training Considering Batch Size

Has LLM Reached the Scaling Ceiling Yet? Unified Insights into LLM Regularities and Constraints

A Survey of Large Language Models

ScalingNote: Scaling up Retrievers with Large Language Models for Real-World Dense Retrieval

Scaling Laws for Downstream Task Performance of Large Language Models

Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models

LLM2: Let Large Language Models Harness System 2 Reasoning

How do Large Language Models Handle Multilingualism?

Large Language Model Enhanced Machine Learning Estimators for Classification

LLM-Generated Natural Language Meets Scaling Laws: New Explorations and Data Augmentation Methods

Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling

Selecting Large Language Model to Fine-tune via Rectified Scaling Law

Observational Scaling Laws and the Predictability of Language Model Performance