Abstract:Code quality evaluation involves scoring generated code quality based on a reference code for a specific problem statement. Currently, there are two main forms of evaluating code quality: match-based evaluation and execution-based evaluation. The former requires the collection of a large number of test cases, making a huge cost. The latter relies on superficial code matching as an evaluation metric, which fails to accurately capture code semantics. Moreover, extensive research has demonstrated that match-based evaluations do not truly reflect code quality. With the development of large language models (LLMs) in recent years, studies have proven the feasibility of using LLMs as evaluators for generative tasks. However, due to issues like hallucinations and uncertainty in LLMs, their correlation with human judgment remains at a lower level, making the direct use of LLMs for code quality evaluation challenging. To address these issues, we propose Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension (HuCoSC). We employ a recursive approach to enable LLMs to comprehend portions of code semantics independently each time, obtaining the code semantics through multiple interactions with LLMs. We designed a Semantic Dependency Decoupling Storage to make independent analysis feasible, allowing LLMs to achieve more accurate semantics by breaking down complex problems. Finally, the generated code is scored based on a semantic comparison between the reference code and itself. Experimental results indicate that HuCoSC surpasses existing state-of-the-art methods in terms of correlation with human experts and correlation with code execution.

Program Comprehension Using Information Retrieval and Probabilistic Finite-State Automata

A Refined-match Method Based on Fuzzy Reasoning

Research on Methodology of Fuzzy Retrieve Based on Soft Computing

A ROUGH-MATCH METHOD BASED ON CHARACTERISTIC SYMBOL IN PROGRAM UNDERSTANDING

Tree-matching Object Concept Assignment to Support Program Comprehension

Goal and Plan Recognition via Parse Trees Using Prefix and Infix Probability Computation

Program Clustering for Comprehension Based on Fuzzy Formal Concept Analysis

Understanding Programs by Exploiting (Fuzzing) Test Cases

An objective-oriented approach to program comprehension using multiple information sources

Automatic Extraction of Software Plans for Program Comprehension Based on Suffix Trees

SPC-FA: synergic parallel compact finite automaton to accelerate multi-string matching with low memory.

Furthest Reasoning with Plan Assessment: Stable Reasoning Path with Retrieval-Augmented Large Language Models

Parallelization and Characterization of Probabilistic Latent Semantic Analysis

Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension

Software Language Comprehension Using a Program-Derived Semantic Graph.

A Language Model for Statements of Software Code

Automaton-Based Representations of Task Knowledge from Generative Language Models

Code Representation Pre-training with Complements from Program Executions

Parallel Absolute-Relative Feature Based Phonotactic Language Recognition.

Bridging The Gap: Entailment Fused-T5 for Open-retrieval Conversational Machine Reading Comprehension

Software Language Comprehension using a Program-Derived Semantics Graph