Abstract:In this paper, we investigate the effectiveness of state-of-the-art LLM, i.e., GPT-4, with three different prompting engineering techniques (i.e., basic prompting, in-context learning, and task-specific prompting) against 18 fine-tuned LLMs on three typical ASE tasks, i.e., code generation, code summarization, and code translation. Our quantitative analysis of these prompting strategies suggests that prompt engineering GPT-4 cannot necessarily and significantly outperform fine-tuning smaller/older LLMs in all three tasks. For comment generation, GPT-4 with the best prompting strategy (i.e., task-specific prompt) had outperformed the first-ranked fine-tuned model by 8.33% points on average in BLEU. However, for code generation, the first-ranked fine-tuned model outperforms GPT-4 with best prompting by 16.61% and 28.3% points, on average in BLEU. For code translation, GPT-4 and fine-tuned baselines tie as they outperform each other on different translation tasks. To explore the impact of different prompting strategies, we conducted a user study with 27 graduate students and 10 industry practitioners. From our qualitative analysis, we find that the GPT-4 with conversational prompts (i.e., when a human provides feedback and instructions back and forth with a model to achieve best results) showed drastic improvement compared to GPT-4 with automatic prompting strategies. Moreover, we observe that participants tend to request improvements, add more context, or give specific instructions as conversational prompts, which goes beyond typical and generic prompting strategies. Our study suggests that, at its current state, GPT-4 with conversational prompting has great potential for ASE tasks, but fully automated prompt engineering with no human in the loop requires more study and improvement.

Improving Formality-Sensitive Machine Translation Using Data-Centric Approaches and Prompt Engineering

Data-Driven Approach for Formality-Sensitive Machine Translation: Language-Specific Handling and Synthetic Data Generation

Machine Translation to Control Formality Features in the Target Language

Controlling Translation Formality Using Pre-trained Multilingual Language Models

A Study on Performance Improvement of Prompt Engineering for Generative AI with a Large Language Model

PromptST: Abstract Prompt Learning for End-to-End Speech Translation

Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning

Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models: A Case Study on ChatGPT.

Refining Translations with LLMs: A Constraint-Aware Iterative Prompting Approach

Prompt Engineering or Fine Tuning: An Empirical Assessment of Large Language Models in Automated Software Engineering Tasks

Minstrel: Structural Prompt Generation with Multi-Agents Coordination for Non-AI Experts

Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT's Customizability

Prompt-Driven Neural Machine Translation

Bridging the Domain Gap: Improve Informal Language Translation Via Counterfactual Domain Adaptation

Recent Advances in Direct Speech-to-text Translation

Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator

Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation

Prompting Large Language Model for Machine Translation: A Case Study

Improved Data Augmentation for Translation Suggestion

Translation of Multifaceted Data without Re-Training of Machine Translation Systems

Exploring Human-Like Translation Strategy with Large Language Models