Defect Identification, Categorization, and Repair:Better Together

Chuanfa Ni,Kaiwen Yang,Xinhui Xia,David Lo,Xiang Chen,Xiaohu Yang

DOI: https://doi.org/10.5281/zenodo.5353354

2021-01-01

Abstract:CompDefect-replication-package This repository contains source code that we used to perform experiments in "Defect Identification, Categorization, and Repair: Better Together" paper. The paper is now under view.

What problem does this paper attempt to address?

Unifying Defect Prediction, Categorization, and Repair by Multi-Task Deep Learning

Chao Ni,Kaiwen Yang,Yan Zhu,Xiang Chen,Xiaohu Yang

DOI: https://doi.org/10.1109/ase56229.2023.00083

2024-01-01

Abstract:Just-In- Time defect prediction models can identify defect-inducing commits at check-in time and many approaches are proposed with remarkable performance. However, these approaches still have a few limitations which affect their effectiveness and practical usage: (1) partially using semantic information or structure information of code, (2) coarsely providing results to a commit (buggy or clean), and (3) independently investigating the defect prediction model and defect repair model. In this study, to handle the aforementioned limitations, we propose a unified defect prediction and repair framework named COMPDEFECT,which can identify whether a changed function inside a commit is defect-prone, categorize the type of defect, and repair such a defect automatically if it falls into several scenarios, e.g., defects with single statement fixes, or those that match a small set of defect templates. Technically, the first two tasks in COMPDEFECT are treated as a multiclass classification task, while the last task is treated as a sequence generation task. To verify the effectiveness of COMPDEFECT, we first build a large-scale function-level dataset (i.e., 21,047) named Function-SStuBs4J and then compare COMPDEFECT with tens of state-of-the-art (SOTA) approaches by considering five performance measures. The experimental results indicate that COMPDEFECT outperforms all SOTAs with a substantial improvement in three tasks separately. Moreover, the pipeline experimental results also indicate the feasibility of COMPDEFECT to unify three tasks in a model.
Defect Identification, Categorization, and Repair: Better Together

Chao Ni,Kaiwen Yang,Xin Xia,David Lo,Xiang Chen,Xiaohu Yang

DOI: https://doi.org/10.48550/arXiv.2204.04856

2022-04-11

Software Engineering

Abstract:Just-In-Time defect prediction (JIT-DP) models can identify defect-inducing commits at check-in time. Even though previous studies have achieved a great progress, these studies still have the following limitations: 1) useful information (e.g., semantic information and structure information) are not fully used; 2) existing work can only predict a commit as buggy one or clean one without more information about what type of defect it is; 3) a commit may involve changes in many files, which cause difficulty in locating the defect; 4) prior studies treat defect identification and defect repair as separate tasks, none aims to handle both tasks simultaneously. In this paper, to handle aforementioned limitations, we propose a comprehensive defect prediction and repair framework named CompDefect, which can identify whether a changed function (a more fine-grained level) is defect-prone, categorize the type of defect, and repair such a defect automatically if it falls into several scenarios, e.g., defects with single statement fixes, or those that match a small set of defect templates. Generally, the first two tasks in CompDefect are treated as a multiclass classification task, while the last one is treated as a sequence generation task. The whole input of CompDefect consists of three parts (exampled with positive functions): the clean version of a function (i.e., the version before defect introduced), the buggy version of a function and the fixed version of a function. In multiclass classification task, CompDefect categorizes the type of defect via multiclass classification with the information in both the clean version and the buggy version. In code sequence generation task, CompDefect repairs the defect once identified or keeps it unchanged.
Just-In-Time Defect Identification and Localization: A Two-Phase Framework.

Meng Yan,Xin Xia,Yuanrui Fan,Ahmed E. Hassan,David Lo,Shanping Li

DOI: https://doi.org/10.1109/tse.2020.2978819

IF: 7.4

2022-01-01

IEEE Transactions on Software Engineering

Abstract:Defect localization aims to locate buggy program elements (e.g., buggy files, methods or lines of code) based on defect symptoms, e.g., bug reports or program spectrum. However, when we receive the defect symptoms, the defect has been exposed and negative impacts have been introduced. Thus, one challenging task is: whether we can locate buggy program prior to the appearance of the defect symptom (e.g., when buggy program elements are being committed to a version control system). We refer to this type of defect localization as “Just-In-Time (JIT) Defect localization”. Although many prior studies have proposed various JIT defect identification methods to identify whether a new change is buggy, these prior methods do not locate the suspicious positions. Thus, JIT defect localization is the next step of JIT defect identification (i.e., after a buggy change is identified, suspicious source code lines are located). To address this problem, we propose a two-phase framework, i.e., JIT defect identification and JIT defect localization. Given a new change, JIT defect identification will identify it as buggy change or clean change first. If a new change is identified as buggy, JIT defect localization will rank the source code lines introduced by the new change according to their suspiciousness scores. The source code lines ranked at the top of the list are estimated as the defect location. For JIT defect identification phase, we use 14 change-level features to build a classifier by following existing approach. For JIT defect localization phase, we propose a JIT defect localization approach that leverages software naturalness with the N-gram model. To evaluate the proposed framework, we conduct an empirical study on 14 open source projects with a total of 177,250 changes. The results show that software naturalness is effective for our JIT defect localization. Our model achieves a reasonable performance, and outperforms the two baselines (i.e., random guess and a static bug finder (i.e., PMD)) by a substantial margin in terms of four ranking measures.
Where Were the Repair Ingredients for Defects4j Bugs?

Yang Deheng,Ministry of Industry and Information Technology,Kim Dongsun,Koyuncu Anil,Kim Kisub,Tian Haoye,Lei Yan,Mao Xiaoguang,Klein Jacques,Bissyandé Tegawendé F.

DOI: https://doi.org/10.1007/s10664-021-10003-7

IF: 3.762

2021-01-01

Empirical Software Engineering

Abstract:A significant body of automated program repair research has built approaches under the redundancy assumption. Patches are then heuristically generated by leveraging repair ingredients (change actions and donor code) that are found in code bases (either the buggy program itself or big code). For example, common change actions (i.e., fix patterns) are frequently mined offline and serve as an important ingredient for many patch generation engines. Although the repetitiveness of code changes has been studied in general, the literature provides little insight into the relationship between the performance of the repair system and the source code base where the change actions were mined. Similarly, donor code is another important repair ingredient to concretize patches guided by abstract patterns. Yet, little attention has been paid to where such ingredients can actually be found. Through a large scale empirical study on the execution results of 24 repair systems evaluated on real-world bugs from Defects4J, we provide a comprehensive view on the distribution of repair ingredients that are relevant for these bugs. In particular, we show that (1) a half of bugs cannot be fixed simply because the relevant repair ingredient is not available in the search space of donor code; (2) bugs that are correctly fixed by literature tools are mostly addressed with shallow change actions; (3) programs with little history of changes can benefit from mining change actions in other programs; (4) parts of donor code to repair a given bug can be found separately at different search locations; (5) bug-triggering test cases are a rich source for donor code search.
Searching for Multi-Fault Programs in Defects4J

Gabin An,Juyeon Yoon,Shin Yoo

DOI: https://doi.org/10.48550/arXiv.2108.04455

2021-08-10

Software Engineering

Abstract:Defects4J has enabled numerous software testing and debugging research work since its introduction. A large part of its contribution, and the resulting popularity, lies in the clear separation and distillation of the root cause of each individual test failure based on careful manual analysis, which in turn allowed researchers to easily study individual faults in isolation. However, in a realistic debugging scenario, multiple faults can coexist and affect test results collectively. Study of automated debugging techniques for these situations, such as failure clustering or fault localisation for multiple faults, would significantly benefit from a reliable benchmark of multiple, coexisting faults. We search for versions of Defects4J subjects that contain multiple faults, by iteratively transplanting fault-revealing test cases across Defects4J versions. Out of 326 studied versions of Defects4J subjects, we report that over 95% (311 versions) actually contain from two to 24 faults. We hope that the extended, multi-fault Defects4J can provide a platform for future research of testing and debugging techniques for multi-fault programs.
A Defect Model Based Testing System

Zhao-hong YANG,Yun-zhan GONG,Qing XIAO,Ya-wen WANG

DOI: https://doi.org/10.3969/j.issn.1007-5321.2008.05.001

2008-01-01

Abstract:The design and implementation of defect testing system (DTS)-a testing system based on defect model was presented. It first classifies, analyzes and summarizes the main software defect models and gives a description method to describe the proposed models based on state machine. A unified testing framework was proposed to test the proposed models. In the unified testing framework, interval computation was proposed to reduce the false positive and provide necessary information for defect testing. The experiment shows that, compared with other tools, DTS can test the code more efficiently but produce less false positive.
Shaping Program Repair Space with Existing Patches and Similar Code

Jiajun Jiang,Yingfei Xiong,Hongyu Zhang,Qing Gao,Xiangqun Chen

DOI: https://doi.org/10.1145/3213846.3213871

2018-01-01

Abstract:Automated program repair (APR) has great potential to reduce bug-fixing effort and many approaches have been proposed in recent years. APRs are often treated as a search problem where the search space consists of all the possible patches and the goal is to identify the correct patch in the space. Many techniques take a data-driven approach and analyze data sources such as existing patches and similar source code to help identify the correct patch. However, while existing patches and similar code provide complementary information, existing techniques analyze only a single source and cannot be easily extended to analyze both. In this paper, we propose a novel automatic program repair approach that utilizes both existing patches and similar code. Our approach mines an abstract search space from existing patches and obtains a concrete search space by differencing with similar code snippets. Then we search within the intersection of the two search spaces. We have implemented our approach as a tool called SimFix, and evaluated it on the Defects4J benchmark. Our tool successfully fixed 34 bugs. To our best knowledge, this is the largest number of bugs fixed by a single technology on the Defects4J benchmark. Furthermore, as far as we know, 13 bugs fixed by our approach have never been fixed by the current approaches.
Can Defects Be Fixed with Weak Test Suites? an Analysis of 50 Defects from Defects4J

Jiajun Jiang,Yingfei Xiong

DOI: https://doi.org/10.48550/arxiv.1705.04149

2017-01-01

Abstract:Automated program repair techniques, which target to generating correct patches for real world defects automatically, have gained a lot of attention in the last decade. Many different techniques and tools have been proposed and developed. However, even the most sophisticated program repair techniques can only repair a small portion of defects while producing a lot of incorrect patches. A possible reason for this low performance is that the test suites of real world programs are usually too weak to guarantee the behavior of the program. To understand to what extent defects can be fixed with weak test suites, we analyzed 50 real world defects from Defects4J, in which we found that up to 84% of them could be correctly fixed. This result suggests that there is plenty of space for current automated program repair techniques to improve. Furthermore, we summarized seven fault localization strategies and seven patch generation strategies that were useful in localizing and fixing these defects, and compared those strategies with current repair techniques. The results indicate potential directions to improve automatic program repair in the future research.
Natural Language Processing Based Detection of Duplicate Defect Patterns

Qian Wu,Qianxiang Wang

DOI: https://doi.org/10.1109/COMPSACW.2010.45

2010-01-01

Abstract:A Defect pattern repository collects different kinds of defect patterns, which are general descriptions of the characteristics of commonly occurring software code defects. Defect patterns can be widely used by programmers, static defect analysis tools, and even runtime verification. Following the idea of web 2.0, defect pattern repositories allow these users to submit defect patterns they found. However, submission of duplicate patterns would lead to a redundancy in the repository. This paper introduces an approach to suggest potential duplicates based on natural language processing. Our approach first computes field similarities based on Vector Space Model, and then employs Information Entropy to determine the field importance, and next combines the field similarities to form the final defect pattern similarity. Two strategies are introduced to make our approach adaptive to special situations. Finally, groups of duplicates are obtained by adopting Hierarchical Clustering. Evaluation indicates that our approach could detect most of the actual duplicates (72% in our experiment) in the repository.
Mining Bug Repositories for Multi-Fault Programs

Dylan Callaghan,Bernd Fischer

2024-04-10

Abstract:Datasets such as Defects4J and BugsInPy that contain bugs from real-world software projects are necessary for a realistic evaluation of automated debugging tools. However these datasets largely identify only a single bug in each entry, while real-world software projects (including those used in Defects4J and BugsInPy) typically contain multiple bugs at the same time. We lift this limitation and describe an extension to these datasets in which multiple bugs are identified in individual entries. We use test case transplantation and fault location translation, in order to expose and locate the bugs, respectively. We thus provide datasets of true multi-fault versions within real-world software projects, which maintain the properties and usability of the original datasets.

Software Engineering
Fault Localization Method of Software Defects Based on Dependencies Analysis of Program Structure

Hui He,Lei Zhao,Qiao Li,Weizhe Zhang,Dongmin Gao,Yongtan Liu

2013-01-01

International Journal of Security and Its Applications

Abstract:Software defects are the major risks of system stable operation. Its error localization technology of automation is one of the key research content for trust computing and software assurance. In this paper, we have proposed a new model, which integrates the current methods by analyzing the program structure. We put forward a new automated fault localization method, this method without the degree loss of automation, at the same time, particle size down to basic positioning code statements, makes the position more accurately. We have designed an experiment to examine the effectiveness of CPD base on SIR. And we proved that the CPD is more effective.
Research of the Defect Model Based on Similarity and Association Rule

Wanjiang Han,Lixin Jiang,Xiaoyan Zhang,Tianbo Lu,Sun Yi,Li Yan,Weijian Li

DOI: https://doi.org/10.11591/telkomnika.v12i6.5452

2014-01-01

Abstract:In order to detect defects efficiently and improve the quality of products, this paper puts forward the concept about defect classification model and defect association model by a lot of defect data. The technology of similarity is applied to defect classification model, and the idea of Knowledge Discovery in Database is applied to defect association model. Defect classification model can analyze the defect efficiently and provides the basis of solving problems quickly while defect association model can be used to detect early and prevent problem, which can make effective improvements to testing and development. This paper summed up GUI defect model based on a large number of interface defects. The model is useful to improve the accuracy of forecast and be used for test planning and implementation through the practice of several projects. DOI : http://dx.doi.org/10.11591/telkomnika.v12i6.5452 Full Text: PDF
Research and implementation of automated C++ code defect detection tool

GU Ke,LIU Chao,JIN Mao-zhong

DOI: https://doi.org/10.3969/j.issn.1001-3695.2009.05.008

2009-01-01

Abstract:The C++ program which is all right in compiling process does not always insure there are no defects in the code.For the reason that there may be defects relative to securities,design and code style,it may result in memory leak,misuse of pointers or make the program code unclearly and unreadable.The defects will place bad impact on the normal running and the maintain ability of the software.This paper introduced a good technology of defect-extendable automated C++ code defect detection,including two methods to detect the defects,a description of defect rules based on XPath technology and an introduction of the C++ defect automation detector.Furthermore,analyzed the detector in stability,credibility and easy-to-use by experiment.
A Manual Inspection of Defects4J Bugs and Its Implications for Automatic Program Repair

Jiajun Jiang,Yingfei Xiong,Xin Xia

DOI: https://doi.org/10.1007/s11432-018-1465-6

2019-01-01

Science China Information Sciences

Abstract:Automatic program repair techniques, which target to generate correct patches for real-world defects automatically, have gained a lot of attention in the last decade. Many different techniques and tools have been proposed and developed. However, even the most sophisticated automatic program repair techniques can only repair a small portion of defects while producing a large number of incorrect patches. A possible reason for the low performance is the test suites of real-world programs are usually too weak to guarantee the behavior of a program. To understand to what extent defects can be fixed with exiting test suites, we manually analyzed 50 real-world defects from Defects4J, where a large portion (i.e., 82%) of them were correctly fixed. This result suggests that there is much room for the current automatic program repair techniques to improve. Furthermore, we summarized seven fault localization and seven patch generation strategies that are useful in localizing and fixing these defects, and compared those strategies with current techniques. The results indicate potential directions to improve automatic program repair in the future.
Unsupervised Defect Prediction Review Protocol

Ning Li,Martin Shepperd,Yuchen Guo

DOI: https://doi.org/10.6084/m9.figshare.7484426.v6

2019-01-01

Abstract:This file is to describe our unsupervised defect prediction review protocol.
Enhancing Redundancy-based Automated Program Repair by Fine-grained Pattern Mining

Jiajun Jiang,Zijie Zhao,Zhirui Ye,Bo Wang,Hongyu Zhang,Junjie Chen

2023-01-01

Abstract:Redundancy-based automated program repair (APR), which generates patches by referencing existing source code, has gained much attention since they are effective in repairing real-world bugs with good interpretability. However, since existing approaches either demand the existence of multi-line similar code or randomly reference existing code, they can only repair a small number of bugs with many incorrect patches, hindering their wide application in practice. In this work, we aim to improve the effectiveness of redundancy-based APR by exploring more effective source code reuse methods for improving the number of correct patches and reducing incorrect patches. Specifically, we have proposed a new repair technique named Repatt, which incorporates a two-level pattern mining process for guiding effective patch generation (i.e., token and expression levels). We have conducted an extensive experiment on the widely-used Defects4J benchmark and compared Repatt with eight state-of-the-art APR approaches. The results show that our approach complements existing approaches by repairing 15 unique bugs compared with the latest deep learning-based methods and 19 unique bugs compared with traditional repair methods when providing the perfect fault localization. In addition, when the perfect fault localization is unknown in real practice, Repatt significantly outperforms the baseline approaches by achieving much higher patch precision, i.e., 83.8%. Moreover, we further proposed an effective patch ranking strategy for combining the strength of Repatt and the baseline methods. The result shows that it repairs 124 bugs when only considering the Top-1 patches and improves the best-performing repair method by repairing 39 more bugs. The results demonstrate the effectiveness of our approach for practical use.
Search-based Duplicate Defect Detection: an Industrial Experience

Mehdi Amoui,Nilam Kaushik,Abraham Al-Dabbagh,Ladan Tahvildari,Shimin Li,Weining Liu

DOI: https://doi.org/10.1109/msr.2013.6624025

2013-01-01

Abstract:Duplicate defects put extra overheads on software organizations, as the cost and effort of managing duplicate defects are mainly redundant. Due to the use of natural language and various ways to describe a defect, it is usually hard to investigate duplicate defects automatically. This problem is more severe in large software organizations with huge defect repositories and massive number of defect reporters. Ideally, an efficient tool should prevent duplicate reports from reaching developers by automatically detecting and/or filtering duplicates. It also should be able to offer defect triagers a list of top-N similar bug reports and allow them to compare the similarity of incoming bug reports with the suggested duplicates. This demand has motivated us to design and develop a search-based duplicate bug detection framework at BlackBerry. The approach follows a generalized process model to evaluate and tune the performance of the system in a systematic way. We have applied the framework on software projects at BlackBerry, in addition to the Mozilla defect repository. The experimental results exhibit the performance of the developed framework and highlight the high impact of parameter tuning on its performance.
MCRepair: Multi-Chunk Program Repair via Patch Optimization with Buggy Block

Jisung Kim,Byeongjung Lee

DOI: https://doi.org/10.48550/arXiv.2310.08157

2023-10-21

Abstract:Automated program repair (APR) is a technology that identifies and repairs bugs automatically. However, repairing multi-chunk bugs remains a long-standing and challenging problem because an APR technique must consider dependencies and then reduce the large patch space. In addition, little is known about how to combine individual candidate patches even though multi-chunk bugs require combinations. Therefore, we propose a novel APR technique called multi-code repair (MCRepair), which applies a buggy block, patch optimization, and CodeBERT to target multi-chunk bugs. A buggy block is a novel method that binds buggy chunks into a multi-buggy chunk and preprocesses the chunk with its buggy contexts for patch space reduction and dependency problems. Patch optimization is a novel strategy that effectively combines the generated candidate patches with patch space reduction. In addition, CodeBERT, a BERT for source code datasets, is fine-tuned to address the lack of datasets and out-of-vocabulary problems. We conducted several experiments to evaluate our approach on six project modules of Defects4J. In the experiments using Defects4J, MCRepair repaired 65 bugs, including 21 multi-chunk bugs. Moreover, it fixed 18 unique bugs, including eight multi-chunk bugs, and improved 40 to 250 percent performance than the baselines.

Software Engineering
Investigating the impact of multiple dependency structures on software defects

Di Cui,Ting Liu,Yuanfang Cai,Qinghua Zheng,Qiong Feng,Wuxia Jin,Jiaqi Guo,Yu Qu

DOI: https://doi.org/10.1109/ICSE.2019.00069

2019-01-01

Abstract:Over the past decades, numerous approaches were proposed to help practitioner to predict or locate defective files. These techniques often use syntactic dependency, history co-change relation, or semantic similarity. The problem is that, it remains unclear whether these different dependency relations will present similar accuracy in terms of defect prediction and localization. In this paper, we present our systematic investigation of this question from the perspective of software architecture. Considering files involved in each dependency type as an individual design space, we model such a design space using one DRSpace. We derived 3 DRSpaces for each of the 117 Apache open source projects, with 643,079 revision commits and 101,364 bug reports in total, and calculated their interactions with defective files. The experiment results are surprising: the three dependency types present significantly different architectural views, and their interactions with defective files are also drastically different. Intuitively, they play completely different roles when used for defect prediction/localization. The good news is that the combination of these structures has the potential to improve the accuracy of defect prediction/localization. In summary, our work provides a new perspective regarding to which type(s) of relations should be used for the task of defect prediction/localization. These quantitative and qualitative results also advance our knowledge of the relationship between software quality and architectural views formed using different dependency types.
Multi-Location Program Repair: Roads Ahead

Shangwen Wang,Xiaoguang Mao,Nan Niu,Xin Yi,Anbang Guo

2018-01-01

Abstract:Automated program repair (APR) has great potential to reduce the effort and time-consumption in software maintenance and becomes a hot topic in software engineering recently with many approaches being proposed. Multi-location program repair has always been a challenge in this field since its complexity in logic and structure. While some approaches do not claim to have the features for solving multi-location bugs, they generate correct patches for these defects in practice. In this paper, we first make an observation on multi-location bugs in Defects4J and divide them into two categories (i.e., similar and relevant multi-location bugs) based on the repair actions in their patches. We then summarize the situation of multi-location bugs in Defects4J fixed by current tools. We analyze the twenty-two patches generated by current tools and propose two feasible strategies for fixing multi-location bugs, illustrating them through two detailed case studies. At last, preliminary results prove the feasibility of our methods with the repair of two bugs that have never been fixed before. By learning from successful experience in the past, this paper points out possible ways ahead for multi-location program repair.

Defect Identification, Categorization, and Repair:Better Together

Unifying Defect Prediction, Categorization, and Repair by Multi-Task Deep Learning

Defect Identification, Categorization, and Repair: Better Together

Just-In-Time Defect Identification and Localization: A Two-Phase Framework.

Where Were the Repair Ingredients for Defects4j Bugs?

Searching for Multi-Fault Programs in Defects4J

A Defect Model Based Testing System

Shaping Program Repair Space with Existing Patches and Similar Code

Can Defects Be Fixed with Weak Test Suites? an Analysis of 50 Defects from Defects4J

Natural Language Processing Based Detection of Duplicate Defect Patterns

Mining Bug Repositories for Multi-Fault Programs

Fault Localization Method of Software Defects Based on Dependencies Analysis of Program Structure

Research of the Defect Model Based on Similarity and Association Rule

Research and implementation of automated C++ code defect detection tool

A Manual Inspection of Defects4J Bugs and Its Implications for Automatic Program Repair

Unsupervised Defect Prediction Review Protocol

Enhancing Redundancy-based Automated Program Repair by Fine-grained Pattern Mining

Search-based Duplicate Defect Detection: an Industrial Experience

MCRepair: Multi-Chunk Program Repair via Patch Optimization with Buggy Block

Investigating the impact of multiple dependency structures on software defects

Multi-Location Program Repair: Roads Ahead