Abstract:Large Language Models (LLMs) perform well on basic programming problems. However, they encounter challenges when dealing with complex tasks involving the use of diverse algorithmic and data structure skills, particularly programming competition-level problems. Notably, ChatGPT exhibits proficient performance on problems it has encountered during its pre-training phase, but this performance deteriorates when faced with novel problems. Consequently, enhancing the ability of LLMs to address unfamiliar problems has emerged as a pivotal research focus. The problem-solving process of LLMs mirrors human programmers' approach to a certain extent. When confronted with new programming tasks, human programmers engage in task planning and code writing with the previously acquired knowledge about algorithms and data structures. Despite having learned such knowledge, LLMs struggle to effectively apply it when faced with specific new problems. To address this issue, we constructed a novel dataset, CodeF, which contains a portion of programming problems that ChatGPT has not previously encountered. Furthermore, we developed a Knowledge Library tailored for Python programming contest problems and introduced the concept of Knowledge-Aware Code Generation (KareCoder). KareCoder bolsters the models' understanding and problem-solving capabilities by integrating prompt and knowledge from the library into the LLMs' code generation reasoning process, especially on Pass@1 metrics. Upon testing on the CodeF and APPS datasets, KareCoder demonstrated outstanding performance in handling novel problems previously unencountered by LLMs. In contrast with the code directly generated by ChatGPT, KareCoder achieved a relative improvement of 23.3% on the Pass@1 metric on the CodeF post2021-9 dataset. Additionally, it performs well compared to other methods when dealing with problems that LLMs have previously encountered.

CodeKGC: Code Language Model for Generative Knowledge Graph Construction

CodeKGC: Code Language Model for Generative Knowledge Graph Construction

Generative Knowledge Graph Construction: A Review

KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion

CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation

Schema-adaptable Knowledge Graph Construction

A Comprehensive Study on Knowledge Graph Embedding over Relational Patterns Based on Rule Learning

A Toolkit for Generating Code Knowledge Graphs

Making Large Language Models Perform Better in Knowledge Graph Completion

Unsupervised Construction of Knowledge Graphs From Text and Code

From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer

Code Generation with Hybrid of Structural and Semantic Features Retrieval

Chain of Knowledge: A Framework for Grounding Large Language Models with Structured Knowledge Bases

Learning Structured Embeddings of Knowledge Graphs with Generative Adversarial Framework

Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generation

Graph Structure Enhanced Pre-Training Language Model for Knowledge Graph Completion

Knowledge-Aware Code Generation with Large Language Models

Language Models of Code are Few-Shot Commonsense Learners

Bridge: A Unified Framework to Knowledge Graph Completion via Language Models and Knowledge Representation