Abstract:The introduction of generative pre-trained models, like GPT-4, has introduced a phenomenon known as prompt engineering, whereby model users repeatedly write and revise prompts while trying to achieve a task. Using these AI models for intelligent features in software applications require using APIs that are controlled through developer-written prompts. These prompts have powered AI experiences in popular software products, potentially reaching millions of users. Despite the growing impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some forms of prompts are programs, and that the development of prompts is a distinct phenomenon in programming. We refer to this phenomenon as prompt programming. To this end, we develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt complexities. Through this study, we contribute 14 observations about prompt programming. For example, rather than building mental models of code, prompt programmers develop mental models of the FM's behavior on the prompt and its unique qualities by interacting with the model. While prior research has shown that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts, each with many iterations, still struggle to develop reliable mental models. This contributes to a rapid and unsystematic development process. Taken together, our observations indicate that prompt programming is significantly different from traditional software development, motivating the creation of tools to support prompt programming. Our findings have implications for software engineering practitioners, educators, and researchers.

The Art and Science of Prompt Engineering: A New Literacy in the Information Age

Prompt Engineering with ChatGPT: A Guide for Academic Writers

Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering

Prompt Engineering: a methodology for optimizing interactions with AI-Language Models in the field of engineering

Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial

A Brief History of Prompt: Leveraging Language Models. (Through Advanced Prompting)

Prompt Engineering For Students of Medicine and Their Teachers

Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts

The Art of Creative Inquiry—From Question Asking to Prompt Engineering

What is prompt literacy? An exploratory study of language learners' development of new literacy skill using generative AI

The Prompt Canvas: A Literature-Based Practitioner Guide for Creating Effective Prompts in Large Language Models

Prompting meaning: a hermeneutic approach to optimising prompt engineering with ChatGPT

Prompt Engineering a Prompt Engineer

Exploring Prompt Engineering: A Systematic Review with SWOT Analysis

The art of prompts' formulation: limitations, potential, and practical examples in large language models

Conversational Prompt Engineering

Aspartate transcarbamylase. Studies of the catalytic subunit by ultraviolet difference spectroscopy.

A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications

Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study

Exploring Prompt Engineering Practices in the Enterprise

Towards a Catalog of Prompt Patterns to Enhance the Discipline of Prompt Engineering