Abstract:Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.
What problem does this paper attempt to address?
The problem this paper attempts to address is: how to optimize the performance of large language models (LLMs) in few-shot in-context learning (ICL) by retrieving demonstrations specific to the input query. Specifically, traditional ICL methods typically use a fixed set of examples, which may not cater to the specific needs of each query, thereby affecting the model's performance. To solve this issue, the paper explores a retrieval-based ICL (RetICL) approach, which dynamically selects the most relevant examples for each input query to enhance the model's adaptability and performance.
### Main Issues of the Paper:
1. **Limitations of Fixed Example Sets**: Traditional ICL methods use fixed example sets that may not adapt well to the specific needs of each query, leading to decreased performance.
2. **Factors Influencing Example Selection**: The selection of examples is influenced by various factors, including the number, format, order, and diversity of examples, all of which affect the model's prediction accuracy.
3. **Advantages of Dynamic Example Selection**: Retrieval-based ICL methods can improve model performance and robustness by dynamically selecting the most relevant examples for the current query.
### Solutions:
- **Retrieval Strategies**: The paper explores various retrieval strategies, including one-hoc retrieval, clustering retrieval, and iterative retrieval, to select the most appropriate examples.
- **Retrieval Objectives**: The paper discusses two main retrieval objectives: similarity and diversity. Similarity refers to selecting examples most similar to the query, while diversity involves selecting different types of examples to cover more query features.
- **Retrieval Corpora**: The paper analyzes different methods for constructing retrieval corpora, including in-domain, mix-domain, and cross-domain settings, to suit different application scenarios.
### Main Contributions:
- **Systematic Review**: The paper provides a comprehensive review of existing retrieval-based ICL research, comparing different design choices, training methods, and inference algorithms.
- **Theoretical and Empirical Analysis**: The paper offers theoretical explanations and validates the effectiveness of retrieval-based ICL methods through extensive experimental results.
- **Future Research Directions**: The paper identifies future research directions for retrieval-based ICL methods, providing references for subsequent studies.
In summary, this paper aims to optimize the performance of large language models in few-shot in-context learning by dynamically selecting the most relevant examples for the query, thereby enhancing the model's adaptability and robustness.