Abstract:Large Language Models (LLMs) perform well on basic programming problems. However, they encounter challenges when dealing with complex tasks involving the use of diverse algorithmic and data structure skills, particularly programming competition-level problems. Notably, ChatGPT exhibits proficient performance on problems it has encountered during its pre-training phase, but this performance deteriorates when faced with novel problems. Consequently, enhancing the ability of LLMs to address unfamiliar problems has emerged as a pivotal research focus. The problem-solving process of LLMs mirrors human programmers' approach to a certain extent. When confronted with new programming tasks, human programmers engage in task planning and code writing with the previously acquired knowledge about algorithms and data structures. Despite having learned such knowledge, LLMs struggle to effectively apply it when faced with specific new problems. To address this issue, we constructed a novel dataset, CodeF, which contains a portion of programming problems that ChatGPT has not previously encountered. Furthermore, we developed a Knowledge Library tailored for Python programming contest problems and introduced the concept of Knowledge-Aware Code Generation (KareCoder). KareCoder bolsters the models' understanding and problem-solving capabilities by integrating prompt and knowledge from the library into the LLMs' code generation reasoning process, especially on Pass@1 metrics. Upon testing on the CodeF and APPS datasets, KareCoder demonstrated outstanding performance in handling novel problems previously unencountered by LLMs. In contrast with the code directly generated by ChatGPT, KareCoder achieved a relative improvement of 23.3% on the Pass@1 metric on the CodeF post2021-9 dataset. Additionally, it performs well compared to other methods when dealing with problems that LLMs have previously encountered.

KareCoder: A New Knowledge-Enriched Code Generation System

Knowledge-Aware Code Generation with Large Language Models

JumpCoder: Go Beyond Autoregressive Coder via Online Modification

SelfEvolve: A Code Evolution Framework via Large Language Models

From Misuse to Mastery: Enhancing Code Generation with Knowledge-Driven AI Chaining

A Pair Programming Framework for Code Generation Via Multi-Plan Exploration and Feedback-Driven Refinement

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback

CodeKGC: Code Language Model for Generative Knowledge Graph Construction

BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models

CoCoST: Automatic Complex Code Generation with Online Searching and Correctness Testing

DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

UniCoder: Scaling Code Large Language Model via Universal Code

CoLadder: Supporting Programmers with Hierarchical Code Generation in Multi-Level Abstraction

Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generation

DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning

Iterative Knowledge Distillation through Feedback-Driven Learning Cycles

MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks

AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation