From BERT to GPT-3 Codex: Harnessing the Potential of Very Large Language Models for Data Management

Immanuel Trummer
DOI: https://doi.org/10.14778/3554821.3554896
2023-06-16
Abstract:Large language models have recently advanced the state of the art on many natural language processing benchmarks. The newest generation of models can be applied to a variety of tasks with little to no specialized training. This technology creates various opportunities for applications in the context of data management. The tutorial will introduce participants to basic background on language models, discuss different methods to use language models, and give an overview and short demonstration of available libraries and APIs. Models for generating natural language will be considered as well as models, such as GPT-3 Codex, which complete program code or generate code from natural language instructions. Finally, the tutorial will discuss recent research in the database community that exploits language models in the context of traditional database systems or proposes novel system architectures that are based on them. The tutorial is targeted at database researchers. No prior background on language models is required. The goal of the tutorial is to introduce database researchers to the latest generation of language models, and to their use cases in the domain of data management.
Databases
What problem does this paper attempt to address?
The paper aims to introduce the potential applications of large language models (such as BERT to GPT-3 Codex) in the field of data management. Specifically, the goals of the paper are: 1. **Introduce Background Knowledge**: To introduce database researchers to the latest foundational knowledge of language models, especially the Transformer architecture and its applications in natural language processing (NLP). 2. **Demonstrate Methods**: To discuss how to use these language models and provide practical examples and demonstrations showcasing their capabilities in text generation, code completion, and more. 3. **Provide Tool Resources**: To outline existing libraries and APIs, making it easier for researchers to leverage these models in their research work. 4. **Explore Application Scenarios**: To discuss various applications of language models in the database field, including improving user interfaces of traditional database systems, fact verification, and database tuning. 5. **Inspire New Research Directions**: To inspire database researchers to explore new system architectures and technologies, utilizing language models to solve complex problems in data management. In summary, this paper, in the form of a tutorial, helps researchers in the database field understand and apply the most advanced language model technologies.