Abstract:Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions.

What problem does this paper attempt to address?

The paper primarily explores the development of Large Language Models (LLMs) and their applications and impacts on Natural Language Processing (NLP) tasks. Specifically: 1. **Background Introduction**: The paper first reviews the historical development of language models, from Statistical Language Models (SLM) and Neural Language Models (NLM) to Pre-trained Language Models (PLM), and finally to the current Large Language Models (LLM). This process demonstrates how language models have gradually become more complex and powerful. 2. **Unique Capabilities of LLMs**: Researchers have found that when the parameter scale of language models reaches a certain level, these large models not only improve in performance but also exhibit some capabilities that smaller models do not possess (such as contextual learning ability). Therefore, the term LLM is used to specifically refer to these large-scale language models. 3. **Technological Evolution**: The paper details the technological evolution of LLMs, including the application of the Transformer architecture, training on large-scale datasets, and the demand for computational resources. Additionally, it discusses how to optimize the training effects of LLMs through scaling laws. 4. **Practical Applications**: LLMs have already had a significant impact in various fields, such as dialogue systems (e.g., ChatGPT), search engines (e.g., New Bing), and multimodal task processing (e.g., GPT-4 supporting image input). In summary, this paper aims to provide a comprehensive overview of the research progress of LLMs and to explore their future development directions and challenges.

A Survey of Large Language Models

Large Language Models Meet NLP: A Survey

Large Language Models: A Survey

History, Development, and Principles of Large Language Models-An Introductory Survey

Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

Large language models (LLMs): survey, technical frameworks, and future challenges

A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)

A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine

Efficient Large Language Models: A Survey

A Survey on Large Language Models from Concept to Implementation

A Survey on Evaluation of Large Language Models

A Survey on Evaluation of Large Language ModelsJust Accepted

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

An analysis of large language models: their impact and potential applications

A Survey for Large Language Models in Biomedicine

Unlocking the potential: A comprehensive exploration of large language models in natural language processing

A Survey of Large Language Models in Medicine: Progress, Application, and Challenge

The Rise and Potential of Large Language Model Based Agents: A Survey

Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers