Abstract:Recent advancements in natural language processing (NLP) have catalyzed the development of models capable of generating coherent and contextually relevant responses. Such models are applied across a diverse array of applications, including but not limited to chatbots, expert systems, question-and-answer robots, and language translation systems. Large Language Models (LLMs), exemplified by OpenAI's Generative Pretrained Transformer (GPT), have significantly transformed the NLP landscape. They have introduced unparalleled abilities in generating text that is not only contextually appropriate but also semantically rich. This evolution underscores a pivotal shift towards more sophisticated and intuitive language understanding and generation capabilities within the field. Models based on GPT are developed through extensive training on vast datasets, enabling them to grasp patterns akin to human writing styles and deliver insightful responses to intricate questions. These models excel in condensing text, extending incomplete passages, crafting imaginative narratives, and emulating conversational exchanges. However, GPT LLMs are not without their challenges, including ethical dilemmas and the propensity for disseminating misinformation. Additionally, the deployment of these models at a practical scale necessitates a substantial investment in training and computational resources, leading to concerns regarding their sustainability. ChatGPT, a variant rooted in transformer-based architectures, leverages a self-attention mechanism for data sequences and a reinforcement learning-based human feedback (RLHF) system. This enables the model to grasp long-range dependencies, facilitating the generation of contextually appropriate outputs. Despite ChatGPT marking a significant leap forward in NLP technology, there remains a lack of comprehensive discourse on its architecture, efficacy, and inherent constraints. Therefore, this survey aims to elucidate the ChatGPT model, offering an in-depth exploration of its foundational structure and operational efficacy. We meticulously examine Chat-GPT's architecture and training methodology, alongside a critical analysis of its capabilities in language generation. Our investigation reveals ChatGPT's remarkable aptitude for producing text indistinguishable from human writing, whilst also acknowledging its limitations and susceptibilities to bias. This analysis is intended to provide a clearer understanding of ChatGPT, fostering a nuanced appreciation of its contributions and challenges within the broader NLP field. We also explore the ethical and societal implications of this technology, and discuss the future of NLP and AI. Our study provides valuable insights into the inner workings of ChatGPT, and helps to shed light on the potential of LLMs for shaping the future of technology and society. The approach used as Eco-GPT, with a three-level cascade (GPT-J, J1-G, GPT-4), achieves 73% and 60% cost savings in CaseHold and CoQA datasets, outperforming GPT-4.

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series

A Systematic Evaluation of Large Language Models of Code

mGPT: Few-Shot Learners Go Multilingual

GLM-130B: An Open Bilingual Pre-trained Model

Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models

GPT-Neo for commonsense reasoning -- a theoretical and practical lens

SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks

OPT: Open Pre-trained Transformer Language Models

SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding

Language Models are Few-Shot Learners

NeuGPT: Unified multi-modal Neural GPT

BloombergGPT: A Large Language Model for Finance

Comparative Study of Large Language Model Architectures on Frontier

Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Demystifying ChatGPT: An In-depth Survey of OpenAI's Robust Large Language Models

h2oGPT: Democratizing Large Language Models

"Medium" LMs of Code in the Era of LLMs: Lessons From StackOverflow

LexGPT 0.1: pre-trained GPT-J models with Pile of Law