Abstract:The introduction of generative pre-trained models, like GPT-4, has introduced a phenomenon known as prompt engineering, whereby model users repeatedly write and revise prompts while trying to achieve a task. Using these AI models for intelligent features in software applications require using APIs that are controlled through developer-written prompts. These prompts have powered AI experiences in popular software products, potentially reaching millions of users. Despite the growing impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some forms of prompts are programs, and that the development of prompts is a distinct phenomenon in programming. We refer to this phenomenon as prompt programming. To this end, we develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt complexities. Through this study, we contribute 14 observations about prompt programming. For example, rather than building mental models of code, prompt programmers develop mental models of the FM's behavior on the prompt and its unique qualities by interacting with the model. While prior research has shown that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts, each with many iterations, still struggle to develop reliable mental models. This contributes to a rapid and unsystematic development process. Taken together, our observations indicate that prompt programming is significantly different from traditional software development, motivating the creation of tools to support prompt programming. Our findings have implications for software engineering practitioners, educators, and researchers.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper explores the phenomenon of using natural language prompts in the software development process, known as "prompt engineering." With the introduction of generative pre-trained models (such as GPT-4), developers have started to achieve specific tasks by writing and modifying natural language prompts. These prompts are integrated into code-based applications, bringing smarter functionalities to software products and impacting millions of users. Despite the increasing importance of prompt engineering, there is very limited research on its development process and its relationship with traditional programming. Therefore, the authors adopted the Straussian grounded theory method and conducted interviews with 20 developers from different backgrounds who are involved in prompt development to understand the process of prompt programming and compare it with traditional software development. The study contributed 14 major observations about prompt programming, indicating significant differences between prompt programming and traditional software development, and highlighting the need for new tools and support processes to address this phenomenon. Specifically, this paper aims to: 1. Understand how developers create programs that include natural language prompts. 2. Explore the uniqueness of prompt programming and its relationship with traditional programming activities. 3. Propose the need for new tools and methods tailored to prompt programming.

Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts

A Brief History of Prompt: Leveraging Language Models. (Through Advanced Prompting)

Can Developers Prompt? A Controlled Experiment for Code Documentation Generation

Prompted Software Engineering in the Era of AI Models

Exploring Prompt Engineering Practices in the Enterprise

The Art and Science of Prompt Engineering: A New Literacy in the Information Age

Prompt Problems: A New Programming Exercise for the Generative AI Era

The Prompt Report: A Systematic Survey of Prompting Techniques

Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators

Promptor: A Conversational and Autonomous Prompt Generation Agent for Intelligent Text Entry Techniques

Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances

Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering

The art of prompts' formulation: limitations, potential, and practical examples in large language models

On Meta-Prompting

Towards a Catalog of Prompt Patterns to Enhance the Discipline of Prompt Engineering

Conversational Prompt Engineering

Prompt Engineering for Nurse Educators

Prompt Engineering: a methodology for optimizing interactions with AI-Language Models in the field of engineering

Prompting meaning: a hermeneutic approach to optimising prompt engineering with ChatGPT

Prompt Engineering or Fine Tuning: An Empirical Assessment of Large Language Models in Automated Software Engineering Tasks

The role of interface design on prompt-mediated creativity in Generative AI