Improving the detection of technical debt in Java source code with an enriched dataset

Nam Le Hai,Anh M. T. Bui,Phuong T. Nguyen,Davide Di Ruscio,Rick Kazman

2024-11-08

Abstract:Technical debt (TD) is a term used to describe the additional work and costs that emerge when developers have opted for a quick and easy solution to a problem, rather than a more effective and well-designed, but time-consuming approach. Self-Admitted Technical Debts (SATDs) are a specific type of technical debts that developers intentionally document and acknowledge, typically via textual comments. While these self-admitted comments are a useful tool for identifying technical debts, most of the existing approaches focus on capturing crucial tokens associated with various categories of TD, neglecting the rich information embedded within the source code itself. Recent research has focused on detecting SATDs by analyzing comments embedded in source code, and there has been little work dealing with technical debts contained in the source code. To fill such a gap, in this study, through the analysis of comments and their associated source code from 974 Java projects hosted in the Stack corpus, we curated the first ever dataset of TD identified by code comments, coupled with its associated source code. Through an empirical evaluation, we found out that the comments of the resulting dataset help enhance the prediction performance of state-of-the-art SATD detection models. More importantly, including the classified source code significantly improves the accuracy in predicting various types of technical debt. In this respect, our work is two-fold: (i) We believe that our dataset will catalyze future work in the domain, inspiring various research issues related to the recognition of technical debt; (ii) The proposed classifiers may serve as baselines for other studies on the detection of TD by means of the curated dataset.

Software Engineering

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to solve the problem of detecting Technical Debt (TD) in Java source code, especially improving the ability to detect Self - Admitted Technical Debt (SATD) by enriching the data set. Specifically, existing methods mainly rely on extracting key tokens from comments in the source code to identify technical debt, but these methods often overlook the rich information contained in the source code itself. To fill this gap, this paper constructs the first technical debt data set identified by code comments by analyzing the comments of 974 Java projects in the Stack corpus and their associated source code, and combines the classified source code. Through empirical evaluation, the study finds that the comments in this data set help improve the prediction performance of existing SATD detection models, and the inclusion of classified source code significantly improves the accuracy of predicting various types of technical debt.

### Main contributions

1. **Comprehensive data processing pipeline**: From data extraction to annotation, it aims to improve annotation efficiency by selecting informative examples, thereby enriching the existing corpus. This pipeline can be iteratively executed when resources and computing power permit to continuously improve the quality of the data set.
2. **New data set (TESORO)**: Specifically used for detecting technical debt in source code. In addition to the existing corpus, TESORO also provides an important new feature, that is, the source code containing debt, which helps to explore a wider range of scenarios to promote the detection of technical debt.
3. **Novel method**: Combine source code information to enhance SATD detection, and effectively utilize these contexts by studying the impact of code contexts of different lengths on the model.
4. **Empirical research**: Conduct empirical research on the constructed data set to evaluate its effectiveness in detecting technical debt in source code, laying a new methodological foundation for identifying technical debt in source code.

### Research questions

1. **RQ1**: Do manually classified comments help improve the detection of SATD? The researchers combined the data set collected by da Silva Maldonado et al. with newly classified comments and used four machine - learning models (BERT, RoBERTa, UniXCoder, CodeBERT) for comparison to determine whether the new comments help predict technical debt.
2. **RQ1**: Does including source code help improve the detection of technical debt? Based on RQ1, the researchers added source code and fine - tuned the above four models to identify technical debt in the code, aiming to explore whether the enriched data set (including classified comments and corresponding source code) is beneficial for the detection of technical debt.
3. **RQ3**: What is the accuracy of different pre - trained models when detecting technical debt only from source code? The researchers used multiple machine - learning models to conduct experiments on the collected data set to study the accuracy of these models when classifying debt only from source code, thereby inspiring future research directions.

### Conclusion

By constructing and evaluating the TESORO data set, this study not only improves the accuracy of technical debt detection, but also provides a valuable foundation and tool for future related research.