Abstract:Large language models have demonstrated outstanding performance on a wide range of tasks such as question answering and code generation. On a high level, given an input, a language model can be used to automatically complete the sequence in a statistically-likely way. Based on this, users prompt these models with language instructions or examples, to implement a variety of downstream tasks. Advanced prompting methods can even imply interaction between the language model, a user, and external tools such as calculators. However, to obtain state-of-the-art performance or adapt language models for specific tasks, complex task- and model-specific programs have to be implemented, which may still require ad-hoc interaction. Based on this, we present the novel idea of Language Model Programming (LMP). LMP generalizes language model prompting from pure text prompts to an intuitive combination of text prompting and scripting. Additionally, LMP allows constraints to be specified over the language model output. This enables easy adaption to many tasks while abstracting language model internals and providing high-level semantics. To enable LMP, we implement LMQL(short for Language Model Query Language), which leverages the constraints and control flow from an LMP prompt to generate an efficient inference procedure that minimizes the number of expensive calls to the underlying language model. We show that LMQL can capture a wide range of state-of-the-art prompting methods in an intuitive way, especially facilitating interactive flows that are challenging to implement with existing high-level APIs. Our evaluation shows that we retain or increase the accuracy on several downstream tasks, while also significantly reducing the required amount of computation or cost in the case of pay-to-use APIs (26-85% cost savings).

Tasks People Prompt: A Taxonomy of LLM Downstream Tasks in Software Verification and Falsification Approaches

Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability

TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks

A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks

PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models

Exploring Prompt Engineering Practices in the Enterprise

Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editing

Towards a Catalog of Prompt Patterns to Enhance the Discipline of Prompt Engineering

Towards Goal-oriented Prompt Engineering for Large Language Models: A Survey

An Empirical Categorization of Prompting Techniques for Large Language Models: A Practitioner's Guide

Efficient Prompting Methods for Large Language Models: A Survey

Comparative Analysis of Prompt Strategies for Large Language Models: Single-Task vs. Multitask Prompts

The language of prompting: What linguistic properties make a prompt successful?

Prompting Is Programming: A Query Language for Large Language Models

A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications

Testing prompt engineering methods for knowledge extraction from text

A Survey on Prompting Techniques in LLMs

PromptWizard: Task-Aware Prompt Optimization Framework

A Communication Theory Perspective on Prompting Engineering Methods for Large Language Models

What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering