Abstract:Automatic programming attempts to minimize human intervention in the generation of executable code, and has been a long-standing challenge in the software engineering community. To advance automatic programming, researchers are focusing on three primary directions: (1) code search that reuses existing code snippets from external databases; (2) code generation that produces new code snippets from natural language; and (3) program repair that refines existing code snippets by fixing detected bugs. Despite significant advancements, the effectiveness of state-of-the-art techniques is still limited, such as the usability of searched code and the correctness of generated code. Motivated by the real-world programming process, where developers usually use various external tools to aid their coding processes, such as code search engines and code testing tools, in this work, we propose \toolname{}, an automatic programming framework that leverages recent large language models (LLMs) to integrate the three research areas to address their inherent limitations. In particular, our framework first leverages different code search strategies to retrieve similar code snippets, which are then used to further guide the code generation process of LLMs. Our framework further validates the quality of generated code by compilers and test cases, and constructs repair prompts to query LLMs for generating correct patches. We conduct preliminary experiments to demonstrate the potential of our framework, \eg helping CodeLlama solve 267 programming problems with an improvement of 62.53\%. As a generic framework, \toolname{} can integrate various code search, generation, and repair tools, combining these three research areas together for the first time. More importantly, it demonstrates the potential of using traditional SE tools to enhance the usability of LLMs in automatic programming.

Automated Coding Challenges Assembly Using Pre-trained Programming Language Models

Automated Generation of Computer Graded Unit Testing-Based Programming Assessments for Education

ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course

Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming

Analyzing Novice and Competent Programmers' Problem-Solving Behaviors Using an Automated Evaluation System

AI-assisted programming question generation: Constructing semantic networks of programming knowledge by local knowledge graph and abstract syntax tree

Estimating Difficulty Levels of Programming Problems with Pre-trained Model

Probeable Problems for Beginner-level Programming-with-AI Contests

PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models

Automatic Organization of Programming Resources on the Web

Let's Ask Students About Their Programs, Automatically

Aspects on Finding the Optimal Practical Programming Exercise for MOOCs

AI-Infused Semantic Model to Enrich and Expand Programming Question Generation

Automating Personalized Parsons Problems with Customized Contexts and Concepts

No Man is an Island: Towards Fully Automatic Programming by Code Search, Code Generation and Program Repair

Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education

Automatic Programming: Large Language Models and Beyond

Toward Automated Programming for Robotic Assembly Using ChatGPT

ClassCode: An Interactive Teaching and Learning Environment for Programming Education in Classrooms

ACES: Generating Diverse Programming Puzzles with with Autotelic Generative Models

Automated generators of examples and problems for studying computer algorithms