Evaluating and Improving ChatGPT-Based Expansion of Abbreviations

Yanjie Jiang,Hui Liu,Lu Zhang

2024-10-31

Abstract:Source code identifiers often contain abbreviations. Such abbreviations may reduce the readability of the source code, which in turn hinders the maintenance of the software applications. To this end, accurate and automated approaches to expanding abbreviations in source code are desirable and abbreviation expansion has been intensively investigated. However, to the best of our knowledge, most existing approaches are heuristics, and none of them has even employed deep learning techniques, let alone the most advanced large language models (LLMs). LLMs have demonstrated cutting-edge performance in various software engineering tasks, and thus it has the potential to expand abbreviation automatically. To this end, in this paper, we present the first empirical study on LLM-based abbreviation expansion. Our evaluation results on a public benchmark suggest that ChatGPT is substantially less accurate than the state-of-the-art approach, reducing precision and recall by 28.2\% and 27.8\%, respectively. We manually analyzed the failed cases, and discovered the root causes for the failures: 1) Lack of contexts and 2) Inability to recognize abbreviations. In response to the first cause, we investigated the effect of various contexts and found surrounding source code is the best selection. In response to the second cause, we designed an iterative approach that identifies and explicitly marks missed abbreviations in prompts. Finally, we proposed a post-condition checking to exclude incorrect expansions that violate commonsense. All such measures together make ChatGPT-based abbreviation expansion comparable to the state of the art while avoiding expensive source code parsing and deep analysis that are indispensable for state-of-the-art approaches.

Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to use large - language models (LLMs) such as ChatGPT to automatically expand abbreviations in source code and improve their accuracy and practicality**. Specifically, the paper focuses on the following points:

1. **Problem background**:
- Source - code identifiers often contain abbreviations, which may reduce the readability of the code and thus affect the maintenance of software applications.
- Most of the existing abbreviation - expansion methods are heuristic - based and do not use deep - learning techniques, especially the state - of - the - art large - language models (LLMs). Therefore, it is of great significance to explore the application of LLMs in abbreviation expansion.

2. **Research objectives**:
- Evaluate and improve the ChatGPT - based abbreviation - expansion method.
- Explore the performance of ChatGPT in the abbreviation - expansion task, identify the reasons for its failures, and propose corresponding improvement measures.

3. **Specific problems**:
- **RQ1**: How accurate is ChatGPT in source - code abbreviation expansion?
- The results show that although ChatGPT can correctly expand more than half of the abbreviations, its precision and recall are 28.2% and 27.8% lower than those of the existing best method, respectively.

- **RQ2**: Where and why can't ChatGPT correctly expand abbreviations?
- The analysis finds that the main reasons are the lack of context and the inability to recognize abbreviations.

- **RQ3**: To what extent can the performance of ChatGPT be improved by adding context information of abbreviations in the prompt?
- The experimental results show that adding context information (such as enclosing files, knowledge graphs, and surrounding code) can significantly improve the performance of ChatGPT, and the effect of adding surrounding code is the best in particular.

- **RQ4**: To what extent can the performance of ChatGPT be improved by explicitly marking unrecognized abbreviations?
- Explicitly marking unrecognized abbreviations and asking ChatGPT to expand again can further improve the recall rate.

- **RQ5**: To what extent can the abbreviation - expansion performance of ChatGPT be further improved by post - condition checking?
- Introducing post - condition checking (such as ensuring that the abbreviation is a subsequence of the full name) can avoid some error expansions that do not conform to common sense, thereby improving the precision.

Through this series of studies, the paper aims to make the ChatGPT - based abbreviation - expansion method reach the level equivalent to the existing best method, while avoiding expensive static - code analysis and being able to work in the presence of compilation errors.