Abstract:Pre-trained models of source code have gained widespread popularity in many code intelligence tasks. Recently, with the scaling of the model and corpus size, large language models have shown the ability of in-context learning (ICL). ICL employs task instructions and a few examples as demonstrations, and then inputs the demonstrations to the language models for making predictions. This new learning paradigm is training-free and has shown impressive performance in various natural language processing and code intelligence tasks. However, the performance of ICL heavily relies on the quality of demonstrations, e.g., the selected examples. It is important to systematically investigate how to construct a good demonstration for code-related tasks. In this paper, we empirically explore the impact of three key factors on the performance of ICL in code intelligence tasks: the selection, order, and number of demonstration examples. We conduct extensive experiments on three code intelligence tasks including code summarization, bug fixing, and program synthesis. Our experimental results demonstrate that all the above three factors dramatically impact the performance of ICL in code intelligence tasks. Additionally, we summarize our findings and provide takeaway suggestions on how to construct effective demonstrations, taking into account these three perspectives. We also show that a carefully-designed demonstration based on our findings can lead to substantial improvements over widely-used demonstration construction methods, e.g., improving BLEU-4, EM, and EM by at least 9.90%, 175.96%, and 50.81% on code summarization, bug fixing, and program synthesis, respectively

Comparable Demonstrations Are Important in In-Context Learning: A Novel Perspective on Demonstration Selection

Are Human-generated Demonstrations Necessary for In-context Learning?

Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning

Comparative Analysis of Demonstration Selection Algorithms for LLM In-Context Learning

Revisiting Demonstration Selection Strategies in In-Context Learning

In-Context Learning Demonstration Selection via Influence Analysis

In-Context Compositional Generalization for Large Vision-Language Models

Curriculum Demonstration Selection for In-Context Learning

Strategic Demonstration Selection for Improved Fairness in LLM In-Context Learning

Misconfidence-based Demonstration Selection for LLM In-Context Learning

DemoShapley: Valuation of Demonstrations for In-Context Learning

Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps

Demonstration Notebook: Finding the Most Suited In-Context Learning Example from Interactions

What Makes Good In-context Demonstrations for Code Intelligence Tasks with LLMs?

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning

Dynamic Demonstrations Controller for In-Context Learning

It Takes One to Tango but More Make Trouble? In-Context Training with Different Number of Demonstrations

Not All Demonstration Examples Are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning

Enhancing In-Context Learning via Implicit Demonstration Augmentation

In-Context Learning with Iterative Demonstration Selection

Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process