Abstract:Natural language processing (NLP) has significantly transformed in the last decade, especially in the field of language modeling. Large language models (LLMs) have achieved SOTA performances on natural language understanding (NLU) and natural language generation (NLG) tasks by learning language representation in self-supervised ways. This paper provides a comprehensive survey to capture the progression of advances in language models. In this paper, we examine the different aspects of language models, which started with a few million parameters but have reached the size of a trillion in a very short time. We also look at how these LLMs transitioned from task-specific to task-independent to task-and-language-independent architectures. This paper extensively discusses different pretraining objectives, benchmarks, and transfer learning methods used in LLMs. It also examines different finetuning and in-context learning techniques used in downstream tasks. Moreover, it explores how LLMs can perform well across many domains and datasets if sufficiently trained on a large and diverse dataset. Next, it discusses how, over time, the availability of cheap computational power and large datasets have improved LLM’s capabilities and raised new challenges. As part of our study, we also inspect LLMs from the perspective of scalability to see how their performance is affected by the model’s depth, width, and data size. Lastly, we provide an empirical comparison of existing trends and techniques and a comprehensive analysis of where the field of LLM currently stands.

Challenges and Applications of Large Language Models

Large Language Models and Knowledge Graphs: Opportunities and Challenges

Applications and Challenges for Large Language Models: from Data Management Perspective

Exploring the landscape of large language models: Foundations, techniques, and challenges

Challenges and Contributing Factors in the Utilization of Large Language Models (LLMs)

A Primer on Large Language Models and their Limitations

An Interdisciplinary Outlook on Large Language Models for Scientific Research

Apprentices to Research Assistants: Advancing Research with Large Language Models

A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)

Large Language Models: A Survey

Large Language Models Meet NLP: A Survey

An analysis of large language models: their impact and potential applications

Multilingual Large Language Models and Curse of Multilinguality

Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

Application of large language models in professional fields

Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions

Large Language Models for Software Engineering: Survey and Open Problems

Large Language Models for Social Networks: Applications, Challenges, and Solutions

Large Language Models and Games: A Survey and Roadmap

Large Language Models Demonstrate the Potential of Statistical Learning in Language

Large Language Models for Education: A Survey and Outlook