Abstract:The introduction of generative pre-trained models, like GPT-4, has introduced a phenomenon known as prompt engineering, whereby model users repeatedly write and revise prompts while trying to achieve a task. Using these AI models for intelligent features in software applications require using APIs that are controlled through developer-written prompts. These prompts have powered AI experiences in popular software products, potentially reaching millions of users. Despite the growing impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some forms of prompts are programs, and that the development of prompts is a distinct phenomenon in programming. We refer to this phenomenon as prompt programming. To this end, we develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt complexities. Through this study, we contribute 14 observations about prompt programming. For example, rather than building mental models of code, prompt programmers develop mental models of the FM's behavior on the prompt and its unique qualities by interacting with the model. While prior research has shown that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts, each with many iterations, still struggle to develop reliable mental models. This contributes to a rapid and unsystematic development process. Taken together, our observations indicate that prompt programming is significantly different from traditional software development, motivating the creation of tools to support prompt programming. Our findings have implications for software engineering practitioners, educators, and researchers.

Prompted Software Engineering in the Era of AI Models

Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts

Prompt Engineering a Prompt Engineer

A Brief History of Prompt: Leveraging Language Models. (Through Advanced Prompting)

Prompt Engineering: a methodology for optimizing interactions with AI-Language Models in the field of engineering

Prompt Engineering or Fine Tuning: An Empirical Assessment of Large Language Models in Automated Software Engineering Tasks

The Art and Science of Prompt Engineering: A New Literacy in the Information Age

Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study

Prompt Engineering with ChatGPT: A Guide for Academic Writers

Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering

A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications

Exploring Prompt Engineering Practices in the Enterprise

Conversational Prompt Engineering

The Prompt Report: A Systematic Survey of Prompting Techniques

Prompt Engineering for Nurse Educators

Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?

Participatory prompting: a user-centric research method for eliciting AI assistance opportunities in knowledge workflows

Requirements Engineering using Generative AI: Prompts and Prompting Patterns

Prompt Sapper: LLM-Empowered Software Engineering Infrastructure for AI-Native Services

Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial

PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization