Refactoring to Pythonic Idioms: A Hybrid Knowledge-Driven Approach Leveraging Large Language Models

Zejun Zhang,Zhenchang Xing,Xiaoxue Ren,Qinghua Lu,Xiwei Xu
DOI: https://doi.org/10.1145/3643776
2024-06-06
Abstract:Pythonic idioms are highly valued and widely used in the Python programming community. However, many Python users find it challenging to use Pythonic idioms. Adopting a rule-based approach or LLM-only approach is not sufficient to overcome three persistent challenges of code idiomatization including code miss, wrong detection and wrong refactoring. Motivated by the determinism of rules and adaptability of LLMs, we propose a hybrid approach consisting of three modules. We not only write prompts to instruct LLMs to complete tasks, but we also invoke Analytic Rule Interfaces (ARIs) to accomplish tasks. The ARIs are Python code generated by prompting LLMs to generate code. We first construct a knowledge module with three elements including ASTscenario, ASTcomponent and Condition, and prompt LLMs to generate Python code for incorporation into an ARI library for subsequent use. After that, for any syntax-error-free Python code, we invoke ARIs from the ARI library to extract ASTcomponent from the ASTscenario, and then filter out ASTcomponent that does not meet the condition. Finally, we design prompts to instruct LLMs to abstract and idiomatize code, and then invoke ARIs from the ARI library to rewrite non-idiomatic code into the idiomatic code. Next, we conduct a comprehensive evaluation of our approach, RIdiom, and Prompt-LLM on nine established Pythonic idioms in RIdiom. Our approach exhibits superior accuracy, F1-score, and recall, while maintaining precision levels comparable to RIdiom, all of which consistently exceed or come close to 90% for each metric of each idiom. Lastly, we extend our evaluation to encompass four new Pythonic idioms. Our approach consistently outperforms Prompt-LLM, achieving metrics with values consistently exceeding 90% for accuracy, F1-score, precision, and recall.
Software Engineering
What problem does this paper attempt to address?
This paper focuses on how to convert non-Pythonic coding habits into common and efficient idioms in the Python language. The author found three challenges in relying solely on rule-based methods or large language models (LLMs) to identify and refactor coding idioms: code omission, false detection, and incorrect refactoring. To address these issues, they propose a hybrid knowledge-driven approach that combines the determinism of rules with the adaptability of LLMs. The method consists of three modules: the knowledge module, the extraction module, and the idioms module. The knowledge module builds a knowledge base that includes AST scenarios, AST components, and conditions based on Python code generated by LLMs. The extraction module uses these Analytical Rule Interfaces (ARIs) to extract AST components that meet the conditions. Finally, the idioms module designs prompts to guide LLMs in abstracting and refactoring code. The paper demonstrates the superiority of this approach, called RIdiom, in comprehensive evaluations on nine known Pythonic idioms. Its accuracy, F1 score, and recall rate all exceed 90%, and it also performs well on four new Pythonic idioms. The research results suggest that this hybrid approach can effectively extend to new Pythonic idioms, providing new avenues for automated conversion of coding idioms.