Abstract:Large Language Models have revolutionized code generation ability by converting natural language descriptions into executable code. However, generating complex code within real-world scenarios remains challenging due to intricate structures, subtle bugs, understanding of advanced data types, and lack of supplementary contents. To address these challenges, we introduce the CoCoST framework, which enhances complex code generation by online searching for more information with planned queries and correctness testing for code refinement. Moreover, CoCoST serializes the complex inputs and outputs to improve comprehension and generates test cases to ensure the adaptability for real-world applications. CoCoST is validated through rigorous experiments on the DS-1000 and ClassEval datasets. Experimental results show that CoCoST substantially improves the quality of complex code generation, highlighting its potential to enhance the practicality of LLMs in generating complex code.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges faced in generating complex code in practical scenarios. Specifically, the paper points out three main challenges: 1. **Offline documents cannot meet the diverse code - generation requirements**: In real - world scenarios, the requirements for code generation often exceed the capabilities of limited offline documents. For example, the problem description may involve functions not covered in pre - collected documents. Moreover, complex code generation usually needs to handle highly complex logic and a series of transformation functions, and simple API examples cannot provide sufficient guidance. 2. **Lack of test cases**: In practical situations, there is often a lack of test cases for automatic code generation. Most existing work relies on pre - existing test cases in datasets, but it is very difficult to directly obtain these test cases in practical scenarios. 3. **Hidden errors in complex code need to be carefully identified and corrected**: Current technologies usually enhance code by analyzing execution errors, but in the case of complex code, even if the code can be executed, it may contain hidden errors. To address these challenges, the paper proposes a new framework named CoCoST (Automatic Complex Code Generation with Online Search and Correctness Testing). CoCoST improves the generation and optimization of complex code through the following methods: - **Online search**: Build LLM prompts by querying web search engines and extracting relevant information, thereby obtaining the latest code patterns and solutions. - **Test case generation**: Use LLM to automatically generate test cases to ensure the correctness of the code. - **Correctness testing**: During the optimization process, not only check for errors, but also verify the correctness of the execution results, and improve LLM's understanding ability of complex data by serializing input and output data types. Through these methods, CoCoST has been experimentally verified on two complex code - generation datasets (DS - 1000 and ClassEval), and the results show that CoCoST significantly improves the quality of complex code generation.

CoCoST: Automatic Complex Code Generation with Online Searching and Correctness Testing

Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback

CodeCoT and Beyond: Learning to Program and Test Like a Developer.

CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generation

COCO: Testing Code Generation Systems via Concretized Instructions

Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generation

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

UniCoder: Scaling Code Large Language Model via Universal Code

Tree-of-Code: A Tree-Structured Exploring Framework for End-to-End Code Generation and Execution in Complex Task Handling

CompCodeVet: A Compiler-guided Validation and Enhancement Approach for Code Dataset

What's Wrong with Your Code Generated by Large Language Models? An Extensive Study

StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback

Improving Natural Language Capability of Code Large Language Model

An Empirical Study on Self-correcting Large Language Models for Data Science Code Generation

ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code

Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models

Code Generation with Hybrid of Structural and Semantic Features Retrieval

VeCoGen: Automating Generation of Formally Verified C Code with Large Language Models

Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation

Code Generation Approach Supporting Complex System Modeling Based on Graph Pattern Matching

Enhancing Repository-Level Code Generation with Integrated Contextual Information