APE: Active Learning-based Tooling for Finding Informative Few-shot Examples for LLM-based Entity Matching

Kun Qian,Yisi Sang,Farima Fatahi Bayat,Anton Belyi,Xianqi Chu,Yash Govind,Samira Khorshidi,Rahul Khot,Katherine Luna,Azadeh Nikfarjam,Xiaoguang Qi,Fei Wu,Xianhan Zhang,Yunyao Li

2024-07-30

Abstract:Prompt engineering is an iterative procedure often requiring extensive manual effort to formulate suitable instructions for effectively directing large language models (LLMs) in specific tasks. Incorporating few-shot examples is a vital and effective approach to providing LLMs with precise instructions, leading to improved LLM performance. Nonetheless, identifying the most informative demonstrations for LLMs is labor-intensive, frequently entailing sifting through an extensive search space. In this demonstration, we showcase a human-in-the-loop tool called APE (Active Prompt Engineering) designed for refining prompts through active learning. Drawing inspiration from active learning, APE iteratively selects the most ambiguous examples for human feedback, which will be transformed into few-shot examples within the prompt. The demo recording can be found with the submission or be viewed at <a class="link-external link-https" href="https://youtu.be/OwQ6MQx53-Y" rel="external noopener nofollow">this https URL</a>.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively select a small number of highly informative examples (few - shot examples) when using large - language models (LLMs) for specific tasks, so as to improve the performance of LLMs. Specifically, the authors focus on how to automatically select, through the method of active learning, several examples that can best improve the performance of LLMs from a large number of candidate examples during the process of prompt engineering, thereby reducing the workload and time cost of manual example screening. The paper mentions that although LLMs can handle zero - shot tasks without pre - training, their performance can be further improved by providing clear and informative few examples. However, manually selecting these examples from a large number of possible examples is a labor - intensive process, especially when a large amount of data needs to be processed. Therefore, the authors propose a tool named APE (Active Prompt Engineering). Through simple interactions with human users, this tool iteratively identifies the most informative and ambiguous examples and incorporates them as few - shot examples into the prompt to enhance the performance of LLMs. The core of APE lies in its active sampling strategy, that is, through different methods (such as randomness - based sampling and self - consistency - based sampling) to identify which examples LLMs are uncertain about, and then select these examples for human annotation, and finally use them to improve the task - execution ability of LLMs. This method not only improves the performance of LLMs but also greatly reduces the degree of human participation, making prompt engineering more efficient and automated.

APE: Active Learning-based Tooling for Finding Informative Few-shot Examples for LLM-based Entity Matching

APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking

Show Less, Instruct More: Enriching Prompts with Definitions and Guidelines for Zero-Shot NER

Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NER

Does Few-Shot Learning Help LLM Performance in Code Synthesis?

Demonstration of Panda: A Weakly Supervised Entity Matching System

Prompt Selection and Augmentation for Few Examples Code Generation in Large Language Model and its Application in Robotics Control

LEAP: LLM instruction-example adaptive prompting framework for biomedical relation extraction

ToolACE: Winning the Points of LLM Function Calling

Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models

Cost-Effective In-Context Learning for Entity Resolution: A Design Space Exploration

Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions

LLMs Learn Task Heuristics from Demonstrations: A Heuristic-Driven Prompting Strategy for Document-Level Event Argument Extraction

Sample Design Engineering: An Empirical Study of What Makes Good Downstream Fine-Tuning Samples for LLMs

LLMs in Biomedicine: A study on clinical Named Entity Recognition

Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars

MLLM-DataEngine: An Iterative Refinement Approach for MLLM

OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Prompting

Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies

PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Models

Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering