SuperCoder2.0: Technical Report on Exploring the feasibility of LLMs as Autonomous Programmer

Anmol Gautam,Kishore Kumar,Adarsh Jha,Mukunda NS,Ishaan Bhola

2024-10-27

Abstract:We present SuperCoder2.0, an advanced autonomous system designed to enhance software development through artificial intelligence. The system combines an AI-native development approach with intelligent agents to enable fully autonomous coding. Key focus areas include a retry mechanism with error output traceback, comprehensive code rewriting and replacement using Abstract Syntax Tree (ast) parsing to minimize linting issues, code embedding technique for retrieval-augmented generation, and a focus on localizing methods for problem-solving rather than identifying specific line numbers. The methodology employs a three-step hierarchical search space reduction approach for code base navigation and bug localization:utilizing Retrieval Augmented Generation (RAG) and a Repository File Level Map to identify candidate files, (2) narrowing down to the most relevant files using a File Level Schematic Map, and (3) extracting 'relevant locations' within these files. Code editing is performed through a two-part module comprising CodeGeneration and CodeEditing, which generates multiple solutions at different temperature values and replaces entire methods or classes to maintain code integrity. A feedback loop executes repository-level test cases to validate and refine solutions. Experiments conducted on the SWE-bench Lite dataset demonstrate SuperCoder2.0's effectiveness, achieving correct file localization in 84.33% of cases within the top 5 candidates and successfully resolving 34% of test instances. This performance places SuperCoder2.0 fourth globally on the SWE-bench leaderboard. The system's ability to handle diverse repositories and problem types highlights its potential as a versatile tool for autonomous software development. Future work will focus on refining the code editing process and exploring advanced embedding models for improved natural language to code mapping.

Software Engineering,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to enhance the level of automation in the software development process by developing an advanced autonomous system named SuperCoder2.0. Specifically, SuperCoder2.0 aims to achieve fully autonomous programming capabilities by combining AI - native development methods with intelligent agent technologies. The following are the main focuses of this system:

1. **Retry Mechanism and Error Output Backtracking**: By introducing advanced retry mechanisms and error output backtracking functions, debugging time is reduced and the robustness of the system is improved. These mechanisms can actively manage errors, automatically retry, and learn from previous errors to improve the performance of the AI model.

2. **Comprehensive Code Rewriting and Replacement**: Use Abstract Syntax Tree (AST) parsing technology to comprehensively rewrite and replace code to minimize code linting problems. This method ensures the consistency and integrity of the code and avoids fragmentation and inconsistency problems caused by code snippet updates.

3. **Code Embedding Technology**: Adopt code embedding technology for Retrieval - Augmented Generation (RAG) to provide more relevant and accurate code suggestions. This technology obtains more context - related retrieval results through natural language input, surpassing traditional embedding methods.

4. **Localization of Problem - Solving Methods**: Different from traditional debugging methods, SuperCoder2.0 focuses on method - level localization rather than specific line numbers. This method improves the ability to understand and solve complex coding problems through a higher level of abstraction.

To achieve the above goals, SuperCoder2.0 adopts a three - step hierarchical search space reduction method to navigate the codebase and locate errors:

1. **Identify Candidate Files Using RAG and Repository File - Level Mapping**: Retrieve the most relevant files from the codebase through RAG technology and further determine candidate files using repository file - level mapping.

2. **Narrow Down to the Most Relevant Files Using File - Level Structure Diagrams**: Create a structure diagram for each file, providing a detailed description of the internal structure of the file, further narrowing the search scope.

3. **Extract "Relevant Locations" in the File**: Precisely locate the specific areas that need to be modified in the determined files.

In addition, code editing is completed through two - part modules: the CodeGeneration module and the CodeEditing module. These two modules work together to generate multiple solutions and make selections at different temperature values, and finally replace the entire method or class to maintain the integrity of the code.

Experimental results show that SuperCoder2.0 performs well on the SWE - bench Lite dataset, being able to correctly locate the top 5 candidate files in 84.33% of cases and successfully solve 34% of the test instances. This indicates that SuperCoder2.0 is competitive on the global SWE - bench ranking list, demonstrating its potential as a multi - purpose tool in autonomous software development. Future work will focus on optimizing the code editing process and exploring more advanced embedding models to improve the natural language - to - code mapping ability.