Abstract:The capabilities of Large Language Models (LLMs) in code generation, particularly for implementing target functionalities from natural language descriptions, have been extensively studied. As an alternative form of natural language, input-output examples (I/O examples) provide an accessible, unambiguous, and flexible way to describe functionalities, but the diversity, sparseness, and incompleteness of I/O examples also place challenges on understanding and implementing requirements. Therefore, generating code from input-output examples (i.e., example-based code generation) provides a new perspective, allowing us to evaluate LLMs' capability to infer target functionalities from limited information and to process new-form requirements. However, related research about LLMs in example-based code generation remains largely unexplored. To fill this gap, this paper presents the first comprehensive study on example-based code generation using LLMs. To address the incorrectness caused by the incompleteness of I/O examples, we adopt an iterative evaluation framework and formalize the objective of example-based code generation as two sequential sub-objectives: generating code conforming to given examples and generating code that successfully implements the target functionalities from (iteratively) given examples. We assess six state-of-the-art LLMs using a new benchmark of 168 diverse target functionalities. The results demonstrate that when requirements were described using iterative I/O examples rather than natural language, the LLMs' score decreased by over 60%, indicating that example-based code generation remains challenging for the evaluated LLMs. More interestingly, the vast majority (even over 95%) of successfully implemented functionalities are achieved in the first round of iterations, suggesting that the LLMs struggle to effectively utilize the iteratively supplemented requirements.

Using a Feedback Loop for LLM-based Infrastructure as Code Generation

A Survey of using Large Language Models for Generating Infrastructure as Code

Generating Feedback-Ladders for Logical Errors in Programming using Large Language Models

Improving Code Generation by Training with Natural Language Feedback

The First Prompt Counts the Most! An Evaluation of Large Language Models on Iterative Example-based Code Generation

Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences

On the Opportunities of Large Language Models for Programming Process Data

Evaluating Language Models for Generating and Judging Programming Feedback

UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback

Training LLMs for Generating IEC 61131-3 Structured Text with Online Feedback

Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement

MTLLM: LLMs are Meaning-Typed Code Constructs

When LLM-based Code Generation Meets the Software Development Process

AI-powered Code Review with LLMs: Early Results

CoLadder: Supporting Programmers with Hierarchical Code Generation in Multi-Level Abstraction

LLMs for Code: The Potential, Prospects, and Problems

IterGen: Iterative Structured LLM Generation

Agent-Driven Automatic Software Improvement

Applying RLAIF for Code Generation with API-usage in Lightweight LLMs

Large Language Models for Tuning Evolution Strategies