Evaluating Prompt Engineering as a Creative Practice

Charlotte Bird

Abstract:This short paper offers an overview of computational creativity evaluation methodologies that can be employed for the evaluation of prompt engineering. This task hopes to spark conversation around the role of computational creativity research in the new world of generative deep learning, and vice versa.

Computer Science,Engineering

What problem does this paper attempt to address?

Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering

Jonas Oppenlaender,Rhema Linder,Johanna Silvennoinen

2024-07-04

Abstract:We are witnessing a novel era of creativity where anyone can create digital content via prompt-based learning (known as prompt engineering). This paper investigates prompt engineering as a novel creative skill for creating AI art with text-to-image generation. In three consecutive studies, we explore whether crowdsourced participants can 1) discern prompt quality, 2) write prompts, and 3) refine prompts. We find that participants could evaluate prompt quality and crafted descriptive prompts, but they lacked style-specific vocabulary necessary for effective prompting. This is in line with our hypothesis that prompt engineering is a new type of skill that is non-intuitive and must first be acquired (e.g., through means of practice and learning) before it can be used. Our studies deepen our understanding of prompt engineering and chart future research directions. We conclude by envisioning four potential futures for prompt engineering.

Human-Computer Interaction
The Art of Creative Inquiry—From Question Asking to Prompt Engineering

Gal Sasson Lazovsky,Tuval Raz,Yoed N. Kenett

DOI: https://doi.org/10.1002/jocb.671

2024-06-19

The Journal of Creative Behavior

Abstract:As artificial intelligence and natural language processing methods rapidly develop, communication plays a pivotal role in every‐day interactions. In this theoretical paper, we explore the overlap and commonalities between question‐asking and prompt engineering. While seemingly distinct, these processes share a common foundation in essential skills like creativity, critical thinking, and cognitive flexibility. We contend that prompt engineering, the art of crafting cues for language models, and question‐asking, the skill of formulating inquiries, form a symbiotic relationship. Delving into question complexity through Bloom's taxonomy and diverse types of questions, we propose strategies for not only efficient but also engaging prompt design. Our theoretical contribution emphasizes the dynamic role of creativity in both processes, offering intriguing perspectives on human–machine interactions and advancing our understanding of language models and communication skills.

psychology, educational
The role of interface design on prompt-mediated creativity in Generative AI

Maddalena Torricelli,Mauro Martino,Andrea Baronchelli,Luca Maria Aiello

2024-02-17

Abstract:Generative AI for the creation of images is becoming a staple in the toolkit of digital artists and visual designers. The interaction with these systems is mediated by \emph{prompting}, a process in which users write a short text to describe the desired image's content and style. The study of prompts offers an unprecedented opportunity to gain insight into the process of human creativity. Yet, our understanding of how people use them remains limited. We analyze more than 145,000 prompts from the logs of two Generative AI platforms (Stable Diffusion and Pick-a-Pic) to shed light on how people \emph{explore} new concepts over time, and how their exploration might be influenced by different design choices in human-computer interfaces to Generative AI. We find that users exhibit a tendency towards exploration of new topics over exploitation of concepts visited previously. However, a comparative analysis of the two platforms, which differ both in scope and functionalities, reveals some stark differences. Features diverting user focus from prompting and providing instead shortcuts for quickly generating image variants are associated with a considerable reduction in both exploration of novel concepts and detail in the submitted prompts. These results carry direct implications for the design of human interfaces to Generative AI and raise new questions regarding how the process of prompting should be aided in ways that best support creativity.

Computers and Society,Human-Computer Interaction,Physics and Society
From Prompt Engineering to Prompt Craft

Joseph Lindley,Roger Whitham

DOI: https://doi.org/10.1145/3689050.3704424

2024-11-21

Abstract:This pictorial presents an ongoing research programme comprising three practice-based Design Research projects conducted through 2024, exploring the affordances of diffusion-based AI image generation systems, specifically Stable Diffusion. The research employs tangible and embodied interactions to investigate emerging qualitative aspects of generative AI, including uncertainty and materiality. Our approach leverages the flexibility and adaptability of Design Research to navigate the rapidly evolving field of generative AI. The pictorial proposes the notion of prompt craft as a productive reframing of prompt engineering. This is comprised of two contributions: (1) reflections on the notion of materiality for diffusion-based generative AI and a proposed method for a craft-like navigation of the latent space within generative AI models and (2) discussing interaction design strategies for designing user interfaces informed by these affordances. The outcomes are presented as strong concepts or intermediate knowledge, applicable to various situations and domains.

Human-Computer Interaction
Evaluating Creativity in Computational Co-Creative Systems

Pegah Karimi,Kazjon Grace,Mary Lou Maher,Nicholas Davis

DOI: https://doi.org/10.48550/arXiv.1807.09886

IF: 14.4

2018-07-25

Artificial Intelligence

Abstract:This paper provides a framework for evaluating creativity in co-creative systems: those that involve computer programs collaborating with human users on creative tasks. We situate co-creative systems within a broader context of computational creativity and explain the unique qualities of these systems. We present four main questions that can guide evaluation in co-creative systems: Who is evaluating the creativity, what is being evaluated, when does evaluation occur and how the evaluation is performed. These questions provide a framework for comparing how existing co-creative systems evaluate creativity, and we apply them to examples of co-creative systems in art, humor, games and robotics. We conclude that existing co-creative systems tend to focus on evaluating the user experience. Adopting evaluation methods from autonomous creative systems may lead to co-creative systems that are self-aware and intentional.
PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation

Mohi Reza,Ioannis Anastasopoulos,Shreya Bhandari,Zachary A. Pardos

2024-10-22

Abstract:Involving subject matter experts in prompt engineering can guide LLM outputs toward more helpful, accurate, and tailored content that meets the diverse needs of different domains. However, iterating towards effective prompts can be challenging without adequate interface support for systematic experimentation within specific task contexts. In this work, we introduce PromptHive, a collaborative interface for prompt authoring, designed to better connect domain knowledge with prompt engineering through features that encourage rapid iteration on prompt variations. We conducted an evaluation study with ten subject matter experts in math and validated our design through two collaborative prompt-writing sessions and a learning gain study with 358 learners. Our results elucidate the prompt iteration process and validate the tool's usability, enabling non-AI experts to craft prompts that generate content comparable to human-authored materials while reducing perceived cognitive load by half and shortening the authoring process from several months to just a few hours.

Human-Computer Interaction,Artificial Intelligence
Prompt Engineering with ChatGPT: A Guide for Academic Writers

Louie Giray

DOI: https://doi.org/10.1007/s10439-023-03272-4

IF: 3.8

2023-06-08

Annals of Biomedical Engineering

Abstract:Prompt engineering is a relatively new discipline that refers to the practice of developing and optimizing prompts to effectively utilize large language models, particularly in natural language processing tasks. However, not many writers and researchers are familiar about this discipline. Hence, in this paper, I aim to highlight the significance of prompt engineering for academic writers and researchers, particularly the fledgling, in the rapidly evolving world of artificial intelligence. I also discuss the concepts of prompt engineering, large language models, and the techniques and pitfalls of writing prompts. Here, I contend that by acquiring prompt engineering skills, academic writers can navigate the changing landscape and leverage large language models to enhance their writing process. As artificial intelligence continues to advance and penetrate the arena of academic writing, prompt engineering equips writers and researchers with the essential skills to effectively harness the power of language models. This enables them to confidently explore new opportunities, enhance their writing endeavors, and remain at the forefront of utilizing cutting-edge technologies in their academic pursuits.

engineering, biomedical
Automating Creativity

Ming-Hui Huang,Roland T. Rust

DOI: https://doi.org/10.48550/arxiv.2405.06915

IF: 14.4

2024-05-11

Artificial Intelligence

Abstract:Generative AI (GenAI) has spurred the expectation of being creative, due to its ability to generate content, yet so far, its creativity has somewhat disappointed, because it is trained using existing data following human intentions to generate outputs. The purpose of this paper is to explore what is required to evolve AI from generative to creative. Based on a reinforcement learning approach and building upon various research streams of computational creativity, we develop a triple prompt-response-reward engineering framework to develop the creative capability of GenAI. This framework consists of three components: 1) a prompt model for expected creativity by developing discriminative prompts that are objectively, individually, or socially novel, 2) a response model for observed creativity by generating surprising outputs that are incrementally, disruptively, or radically innovative, and 3) a reward model for improving creativity over time by incorporating feedback from the AI, the creator/manager, and/or the customers. This framework enables the application of GenAI for various levels of creativity strategically.
Generative AI tools in art education: Exploring prompt engineering and iterative processes for enhanced creativity

James Hutson,Peter Cotroneo

DOI: https://doi.org/10.54517/m.v4i1.2164

2023-06-05

Abstract:The rapid development and adoption of generative artificial intelligence (AI) tools in the art and design education landscape have introduced both opportunities and challenges. This timely study addresses the need to effectively integrate these tools into the classroom while considering ethical implications and the importance of prompt engineering. By examining the iterative process of refining original ideas through multiple iterations, verbal expansion, and the use of OpenAI’s DALL-E2 for generating diverse visual outcomes, researchers gain insights into the potential benefits and pitfalls of these tools in an educational context. Students in the digital at case study were taught prompt engineering techniques and were tasked with crafting multiple prompts, focusing on refining their ideas over time. Participants demonstrated an increased understanding of the potential and limitations of generative AI tools and how to manipulate subject matter for more effective results. The iterative process encouraged students to explore and experiment with their creative ideas, leading to a deeper understanding of the possibilities offered by AI tools. Despite acknowledging the ethical concerns regarding copyright and the potential replacement of artists, students appreciated the value of generative AI tools for enhancing their sketchbooks and ideation process. Through prompt engineering and iterative processes, students developed a more detail-oriented approach to their work. The challenge of using AI-generated images as final products was conceptually intriguing, requiring further investigation and consideration of the prompts. This study highlights the potential benefits and challenges of integrating generative AI tools into art and design classrooms, emphasizing the importance of prompt engineering, iterative processes, and ethical considerations as these technologies continue to evolve.

Education,Art,Computer Science
Prompt Engineering: a methodology for optimizing interactions with AI-Language Models in the field of engineering

Juan David Velásquez-Henao,Carlos Jaime Franco-Cardona,Lorena Cadavid-Higuita

DOI: https://doi.org/10.15446/dyna.v90n230.111700

2023-11-03

DYNA

Abstract:ChatGPT is a versatile conversational Artificial Intelligence model that responds to user input prompts, with applications in academia and various sectors. However, crafting effective prompts can be challenging, leading to potentially inaccurate or contextually inappropriate responses, emphasizing the importance of prompt engineering in achieving accurate outcomes across different domains. This study aims to address this void by introducing a methodology for optimizing interactions with Artificial Intelligence language models, like ChatGPT, through prompts in the field of engineering. The approach is called GPEI and relies on the latest advancements in this area; and consists of four steps: define the objective, design the prompt, evaluate the response, and iterate. Our proposal involves two key aspects: data inclusion in prompt design for engineering applications and the integration of Explainable Artificial Intelligence principles to assess responses, enhancing transparency. It combines insights from various methodologies to address issues like hallucinations, emphasizing iterative prompt refinement techniques like posing opposing questions and using specific patterns for improvement. This methodology could improve prompt precision and utility in engineering.

engineering, multidisciplinary
When Teams Embrace AI: Human Collaboration Strategies in Generative Prompting in a Creative Design Task

Yuanning Han,Jiale Cheng,Ziyi Qiu,Ray Lc

DOI: https://doi.org/10.1145/3613904.3642133

2024-05-11

Abstract:Studies of Generative AI (GenAI)-assisted creative workflows have focused on individuals overcoming challenges of prompting to produce what they envisioned. When designers work in teams, how do collaboration and prompting influence each other, and how do users perceive generative AI and their collaborators during the co-prompting process? We engaged students with design or performance backgrounds, and little exposure to GenAI, to work in pairs with GenAI to create stage designs based on a creative theme. We found two patterns of collaborative prompting focused on generating story descriptions first, or visual imagery first. GenAI tools helped participants build consensus in the task, and allowed for discussion of the prompting strategies. Participants perceived GenAI as efficient tools rather than true collaborators, suggesting that human partners reduced the reliance on their use. This work highlights the importance of human-human collaboration when working with GenAI tools, suggesting systems that take advantage of shared human expertise in the prompting process.

Psychology,Computer Science
The Art and Science of Prompt Engineering: A New Literacy in the Information Age

Leo S. Lo

DOI: https://doi.org/10.1080/10875301.2023.2227621

2023-06-24

Internet Reference Services Quarterly

Abstract:The novel discipline of prompt-engineering is a combination of artificial intelligence, linguistics, and user experience design. Crafting effective prompts for AI models like OpenAI's GPT can optimize the quality of generated output, especially in reference services. The article explains the CLEAR Framework—a guiding tool that incorporates principles of conciseness, logic, explicitness, adaptability, and reflectiveness in prompt crafting. Despite challenges including biases, ethical concerns, and keeping up with rapidly evolving AI capabilities, the field also presents opportunities for growth. The article concludes with a call to action for library professionals to embrace and master prompt-engineering as an essential skill.
Towards A Diffractive Analysis of Prompt-Based Generative AI

Nina Rajcic,Maria Teresa Llano,Jon McCormack

DOI: https://doi.org/10.1145/3613904.3641971

2024-03-04

Abstract:Recent developments in prompt-based generative AI has given rise to discourse surrounding the perceived ethical concerns, economic implications, and consequences for the future of cultural production. As generative imagery becomes pervasive in mainstream society, dominated primarily by emerging industry leaders, we encourage that the role of the CHI community be one of inquiry; to investigate the numerous ways in which generative AI has the potential to, and already is, augmenting human creativity. In this paper, we conducted a diffractive analysis exploring the potential role of prompt-based interfaces in artists' creative practice. Over a two week period, seven visual artists were given access to a personalised instance of Stable Diffusion, fine-tuned on a dataset of their work. In the following diffractive analysis, we identified two dominant modes adopted by participants, AI for ideation, and AI for production. We furthermore present a number of ethical design considerations for the future development of generative AI interfaces.

Human-Computer Interaction
Performing Creativity With Computational Tools

Daniel Lopes,Jéssica Parente,Pedro Silva,Licínio Roque,Penousal machado

DOI: https://doi.org/10.48550/arXiv.2103.05533

2021-03-10

Abstract:The introduction of new tools in people's workflow has always been promotive of new creative paths. This paper discusses the impact of using computational tools in the performance of creative tasks, especially focusing on graphic design. The study was driven by a grounded theory methodology, applied to a set of semi-structured interviews, made to twelve people working in the areas of graphic design, data science, computer art, music and data visualisation. Among other questions, the results suggest some scenarios in which it is or it is not worth investing in the development of new intelligent creativity-aiding tools.

Computers and Society
How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models

Hai Dang,Lukas Mecke,Florian Lehmann,Sven Goller,Daniel Buschek

DOI: https://doi.org/10.48550/arXiv.2209.01390

IF: 6.4588

2022-09-03

Human-Computer Interaction

Abstract:Deep generative models have the potential to fundamentally change the way we create high-fidelity digital content but are often hard to control. Prompting a generative model is a promising recent development that in principle enables end-users to creatively leverage zero-shot and few-shot learning to assign new tasks to an AI ad-hoc, simply by writing them down. However, for the majority of end-users writing effective prompts is currently largely a trial and error process. To address this, we discuss the key opportunities and challenges for interactive creative applications that use prompting as a new paradigm for Human-AI interaction. Based on our analysis, we propose four design goals for user interfaces that support prompting. We illustrate these with concrete UI design sketches, focusing on the use case of creative writing. The research community in HCI and AI can take these as starting points to develop adequate user interfaces for models capable of zero- and few-shot learning.
The Creativity of Text-to-Image Generation

Jonas Oppenlaender

DOI: https://doi.org/10.1145/3569219.3569352

2022-10-31

Abstract:Text-guided synthesis of images has made a giant leap towards becoming a mainstream phenomenon. With text-to-image generation systems, anybody can create digital images and artworks. This provokes the question of whether text-to-image generation is creative. This paper expounds on the nature of human creativity involved in text-to-image art (so-called "AI art") with a specific focus on the practice of prompt engineering. The paper argues that the current product-centered view of creativity falls short in the context of text-to-image generation. A case exemplifying this shortcoming is provided and the importance of online communities for the creative ecosystem of text-to-image art is highlighted. The paper provides a high-level summary of this online ecosystem drawing on Rhodes' conceptual four P model of creativity. Challenges for evaluating the creativity of text-to-image generation and opportunities for research on text-to-image generation in the field of Human-Computer Interaction (HCI) are discussed.

Human-Computer Interaction,Graphics
PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation

Yingchaojie Feng,Xingbo Wang,Kam Kwai Wong,Sijia Wang,Yuhong Lu,Minfeng Zhu,Baicheng Wang,Wei Chen

DOI: https://doi.org/10.1109/tvcg.2023.3327168

IF: 5.2

2023-01-01

IEEE Transactions on Visualization and Computer Graphics

Abstract:Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refinement, PromptMagician introduces a multi-level visualization for the cross-modal embedding of the retrieved images and recommended keywords, and supports users in specifying multiple criteria for personalized exploration. Two usage scenarios, a user study, and expert interviews demonstrate the effectiveness and usability of our system, suggesting it facilitates prompt engineering and improves the creativity support of the generative text-to-image model.

computer science, software engineering
Prompt Engineering a Prompt Engineer

Qinyuan Ye,Maxamed Axmed,Reid Pryzant,Fereshte Khani

2024-07-03

Abstract:Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models on customized tasks. It requires complex reasoning to examine the model's errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that large language models can be meta-prompted to perform automatic prompt engineering, we argue that their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. We fill this gap by infusing into the meta-prompt three key components: detailed descriptions, context specification, and a step-by-step reasoning template. The resulting method, named PE2, exhibits remarkable versatility across diverse language tasks. It finds prompts that outperform "let's think step by step" by 6.3% on MultiArith and 3.1% on GSM8K, and outperforms competitive baselines on counterfactual tasks by 6.9%. Further, we show that PE2 can make targeted and highly specific prompt edits, rectify erroneous prompts, and induce multi-step plans for complex tasks.

Computation and Language,Artificial Intelligence,Machine Learning
A Brief History of Prompt: Leveraging Language Models. (Through Advanced Prompting)

Golam Md Muktadir

DOI: https://doi.org/10.48550/arXiv.2310.04438

2023-11-29

Abstract:This paper presents a comprehensive exploration of the evolution of prompt engineering and generation in the field of natural language processing (NLP). Starting from the early language models and information retrieval systems, we trace the key developments that have shaped prompt engineering over the years. The introduction of attention mechanisms in 2015 revolutionized language understanding, leading to advancements in controllability and context-awareness. Subsequent breakthroughs in reinforcement learning techniques further enhanced prompt engineering, addressing issues like exposure bias and biases in generated text. We examine the significant contributions in 2018 and 2019, focusing on fine-tuning strategies, control codes, and template-based generation. The paper also discusses the growing importance of fairness, human-AI collaboration, and low-resource adaptation. In 2020 and 2021, contextual prompting and transfer learning gained prominence, while 2022 and 2023 witnessed the emergence of advanced techniques like unsupervised pre-training and novel reward shaping. Throughout the paper, we reference specific research studies that exemplify the impact of various developments on prompt engineering. The journey of prompt engineering continues, with ethical considerations being paramount for the responsible and inclusive future of AI systems.

Computation and Language,Artificial Intelligence
Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study

David James Woo,Deliang Wang,Tim Yung,Kai Guo

DOI: https://doi.org/10.48550/arxiv.2408.07302

2024-07-30

Computers and Society

Abstract:Prompt engineering is critical for effective interaction with large language models (LLMs) such as ChatGPT. However, efforts to teach this skill to students have been limited. This study designed and implemented a prompt engineering intervention, examining its influence on undergraduate students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts. The intervention involved 27 students who participated in a 100-minute workshop conducted during their history course at a university in Hong Kong. During the workshop, students were introduced to prompt engineering strategies, which they applied to plan the course's final essay task. Multiple data sources were collected, including students' responses to pre- and post-workshop questionnaires, pre- and post-workshop prompt libraries, and written reflections. The study's findings revealed that students demonstrated a higher level of AI self-efficacy, an enhanced understanding of AI concepts, and improved prompt engineering skills because of the intervention. These findings have implications for AI literacy education, as they highlight the importance of prompt engineering training for specific higher education use cases. This is a significant shift from students haphazardly and intuitively learning to engineer prompts. Through prompt engineering education, educators can faciitate students' effective navigation and leverage of LLMs to support their coursework.

Evaluating Prompt Engineering as a Creative Practice

Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering

The Art of Creative Inquiry—From Question Asking to Prompt Engineering

The role of interface design on prompt-mediated creativity in Generative AI

From Prompt Engineering to Prompt Craft

Evaluating Creativity in Computational Co-Creative Systems

PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation

Prompt Engineering with ChatGPT: A Guide for Academic Writers

Automating Creativity

Generative AI tools in art education: Exploring prompt engineering and iterative processes for enhanced creativity

Prompt Engineering: a methodology for optimizing interactions with AI-Language Models in the field of engineering

When Teams Embrace AI: Human Collaboration Strategies in Generative Prompting in a Creative Design Task

The Art and Science of Prompt Engineering: A New Literacy in the Information Age

Towards A Diffractive Analysis of Prompt-Based Generative AI

Performing Creativity With Computational Tools

How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models

The Creativity of Text-to-Image Generation

PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation

Prompt Engineering a Prompt Engineer

A Brief History of Prompt: Leveraging Language Models. (Through Advanced Prompting)

Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study