Skill-Based Few-Shot Selection for In-Context Learning

Shengnan An,Bo Zhou,Zeqi Lin,Qiang Fu,Bei Chen,Nanning Zheng,Weizhu Chen,Jian-Guang Lou

DOI: https://doi.org/10.48550/arXiv.2305.14210

2023-10-11

Abstract:In-context learning is the paradigm that adapts large language models to downstream tasks by providing a few examples. Few-shot selection -- selecting appropriate examples for each test instance separately -- is important for in-context learning. In this paper, we propose Skill-KNN, a skill-based few-shot selection method for in-context learning. The key advantages of Skill-KNN include: (1) it addresses the problem that existing methods based on pre-trained embeddings can be easily biased by surface natural language features that are not important for the target task; (2) it does not require training or fine-tuning of any models, making it suitable for frequently expanding or changing example banks. The key insight is to optimize the inputs fed into the embedding model, rather than tuning the model itself. Technically, Skill-KNN generates the skill-based descriptions for each test case and candidate example by utilizing a pre-processing few-shot prompting, thus eliminating unimportant surface features. Experimental results across five cross-domain semantic parsing datasets and six backbone models show that Skill-KNN significantly outperforms existing methods.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively select appropriate examples in in-context learning. Specifically, existing methods based on pre-trained embeddings are susceptible to superficial natural language features that are not important for the target task. Additionally, existing methods often require training or fine-tuning the model, which is inconvenient in practical applications, especially when the example library is frequently expanded or changed.

To address these issues, the authors propose a skill-based few-shot selection method—SKILL-KNN. The main advantages of this method include:
1. **Reducing Bias**: By optimizing the content input to the embedding model rather than adjusting the model itself, it reduces reliance on unimportant superficial features.
2. **No Training Required**: It does not require training or fine-tuning any model, making it suitable for frequently expanding or changing example libraries.

Technically, SKILL-KNN generates skill descriptions for each test case and candidate example through pre-processed few-shot prompts, thereby eliminating unimportant superficial features. Experimental results show that SKILL-KNN significantly outperforms existing methods on 5 cross-domain semantic parsing datasets and 6 backbone models.