Large Language Models for Data Annotation and Synthesis: A Survey

Zhen Tan,Dawei Li,Song Wang,Alimohammad Beigi,Bohan Jiang,Amrita Bhattacharjee,Mansooreh Karami,Jundong Li,Lu Cheng,Huan Liu
2024-12-03
Abstract:Data annotation and synthesis generally refers to the labeling or generating of raw data with relevant information, which could be used for improving the efficacy of machine learning models. The process, however, is labor-intensive and costly. The emergence of advanced Large Language Models (LLMs), exemplified by GPT-4, presents an unprecedented opportunity to automate the complicated process of data annotation and synthesis. While existing surveys have extensively covered LLM architecture, training, and general applications, we uniquely focus on their specific utility for data annotation. This survey contributes to three core aspects: LLM-Based Annotation Generation, LLM-Generated Annotations Assessment, and LLM-Generated Annotations Utilization. Furthermore, this survey includes an in-depth taxonomy of data types that LLMs can annotate, a comprehensive review of learning strategies for models utilizing LLM-generated annotations, and a detailed discussion of the primary challenges and limitations associated with using LLMs for data annotation and synthesis. Serving as a key guide, this survey aims to assist researchers and practitioners in exploring the potential of the latest LLMs for data annotation, thereby fostering future advancements in this critical field.
Computation and Language
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the challenges existing in the data annotation and synthesis process, especially in the fields of machine learning and natural language processing (NLP). Specifically, data annotation and synthesis are not only time - consuming and costly, but also require domain - specific expertise, which makes manual annotation or the creation of large - scale datasets resource - intensive. Although data annotation and synthesis play a crucial role in the training of machine - learning models, their complexity, subjectivity and diversity pose significant challenges to current machine - learning models. The paper points out that large - language models (LLMs), such as GPT - 4, etc., offer an unprecedented opportunity to automate this complex data - annotation process. These models can not only improve the efficiency and accuracy of data annotation, but also ensure the consistency of large - scale data, and adapt to different application requirements through fine - tuning or domain - specific prompts. Therefore, the main objective of this paper is to explore how to use LLMs for data annotation and synthesis, evaluate the quality of the generated annotations, and explore the application methods of these generated annotations at different stages, including supervised fine - tuning, alignment tuning and inference time, etc. In addition, the paper also discusses the main challenges and limitations faced when using LLMs for data annotation and synthesis, aiming to provide researchers and practitioners with a crucial guide to exploring the potential of the latest LLMs in data annotation, thereby promoting the future development of this field.