Abstract:In this paper, we investigate the effectiveness of state-of-the-art LLM, i.e., GPT-4, with three different prompting engineering techniques (i.e., basic prompting, in-context learning, and task-specific prompting) against 18 fine-tuned LLMs on three typical ASE tasks, i.e., code generation, code summarization, and code translation. Our quantitative analysis of these prompting strategies suggests that prompt engineering GPT-4 cannot necessarily and significantly outperform fine-tuning smaller/older LLMs in all three tasks. For comment generation, GPT-4 with the best prompting strategy (i.e., task-specific prompt) had outperformed the first-ranked fine-tuned model by 8.33% points on average in BLEU. However, for code generation, the first-ranked fine-tuned model outperforms GPT-4 with best prompting by 16.61% and 28.3% points, on average in BLEU. For code translation, GPT-4 and fine-tuned baselines tie as they outperform each other on different translation tasks. To explore the impact of different prompting strategies, we conducted a user study with 27 graduate students and 10 industry practitioners. From our qualitative analysis, we find that the GPT-4 with conversational prompts (i.e., when a human provides feedback and instructions back and forth with a model to achieve best results) showed drastic improvement compared to GPT-4 with automatic prompting strategies. Moreover, we observe that participants tend to request improvements, add more context, or give specific instructions as conversational prompts, which goes beyond typical and generic prompting strategies. Our study suggests that, at its current state, GPT-4 with conversational prompting has great potential for ASE tasks, but fully automated prompt engineering with no human in the loop requires more study and improvement.

A Study on Performance Improvement of Prompt Engineering for Generative AI with a Large Language Model

Prompt Engineering or Fine Tuning: An Empirical Assessment of Large Language Models in Automated Software Engineering Tasks

Towards Goal-oriented Prompt Engineering for Large Language Models: A Survey

Unleashing the Potential of Large Language Models as Prompt Optimizers: An Analogical Analysis with Gradient-based Model Optimizers

SPRIG: Improving Large Language Model Performance by System Prompt Optimization

Autonomous Prompt Engineering in Large Language Models

Automatic Prompt Selection for Large Language Models

RePrompt: Planning by Automatic Prompt Engineering for Large Language Models Agents

GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning

Few-shot is enough: exploring ChatGPT prompt engineering method for automatic question generation in english education

Which is better? Exploring Prompting Strategy For LLM-based Metrics

Deliberate then Generate: Enhanced Prompting Framework for Text Generation

Prompt Space Optimizing Few-shot Reasoning Success with Large Language Models

Prompt-RAG: Pioneering Vector Embedding-Free Retrieval-Augmented Generation in Niche Domains, Exemplified by Korean Medicine

Prompt Engineering a Prompt Engineer

The art of prompts' formulation: limitations, potential, and practical examples in large language models

A Systematic Review on Prompt Engineering in Large Language Models for K-12 STEM Education

Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers

A Communication Theory Perspective on Prompting Engineering Methods for Large Language Models