Conversational Prompt Engineering

Liat Ein-Dor,Orith Toledo-Ronen,Artem Spector,Shai Gretz,Lena Dankin,Alon Halfon,Yoav Katz,Noam Slonim
2024-08-09
Abstract:Prompts are how humans communicate with LLMs. Informative prompts are essential for guiding LLMs to produce the desired output. However, prompt engineering is often tedious and time-consuming, requiring significant expertise, limiting its widespread use. We propose Conversational Prompt Engineering (CPE), a user-friendly tool that helps users create personalized prompts for their specific tasks. CPE uses a chat model to briefly interact with users, helping them articulate their output preferences and integrating these into the prompt. The process includes two main stages: first, the model uses user-provided unlabeled data to generate data-driven questions and utilize user responses to shape the initial instruction. Then, the model shares the outputs generated by the instruction and uses user feedback to further refine the instruction and the outputs. The final result is a few-shot prompt, where the outputs approved by the user serve as few-shot examples. A user study on summarization tasks demonstrates the value of CPE in creating personalized, high-performing prompts. The results suggest that the zero-shot prompt obtained is comparable to its - much longer - few-shot counterpart, indicating significant savings in scenarios involving repetitive tasks with large text volumes.
Computation and Language
What problem does this paper attempt to address?
The paper attempts to address the issue that the process of designing and optimizing prompts in large language models (LLMs) is both time-consuming and complex, requiring a significant amount of expertise. To simplify this process, the paper proposes Conversational Prompt Engineering (CPE), a user-friendly tool designed to help users create personalized prompts through brief conversations. Specifically, CPE leverages chat models to interact with users, helping them clarify their output preferences and integrating these preferences into the prompts. This approach not only reduces the need for annotated data and initial prompts required for designing prompts but also enables even inexperienced users to easily create high-quality, personalized prompts. Furthermore, research shows that zero-shot prompts generated through CPE can perform comparably to multi-example prompts, which offers significant cost-effectiveness when handling a large number of text tasks.