Abstract:Regression testing activities greatly reduce the risk of faulty software release. However, the size of the test suites grows throughout the development process, resulting in time-consuming execution of the test suite and delayed feedback to the software development team. This has urged the need for approaches such as test case prioritization (TCP) and test-suite reduction to reach better results in case of limited resources. In this regard, proposing approaches that use auxiliary sources of data such as bug history can be interesting. We aim to propose an approach for TCP that takes into account test case coverage data, bug history, and test case diversification. To evaluate this approach we study its performance on real-world open-source projects. The bug history is used to estimate the fault-proneness of source code areas. The diversification of test cases is preserved by incorporating fault-proneness on a clustering-based approach scheme. The proposed methods are evaluated on datasets collected from the development history of five real-world projects including 357 versions in total. The experiments show that the proposed methods are superior to coverage-based TCP methods. The proposed approach shows that improvement of coverage-based and fault-proneness-based methods is possible by using a combination of diversification and fault-proneness incorporation.
What problem does this paper attempt to address?
### The problems the paper attempts to solve
This paper attempts to solve the problem of Test Case Prioritization (TCP) in software regression testing. Specifically, as the code base continues to grow during the software development process, regression testing becomes increasingly resource - intensive. To improve the cost - effectiveness of regression testing, researchers have proposed various techniques, including test suite reduction, test case selection, and test case prioritization. This paper focuses specifically on test case prioritization and proposes a new method that combines fault - proneness estimation and test case diversity to optimize the execution order of test cases, thereby detecting defects earlier.
### Background and motivation
1. **The importance of regression testing**:
- Regression testing reduces the risk when software is released by running a set of test cases regularly or after each modification.
- As the code base grows and there are frequent code commits, regression testing becomes very time - consuming, resulting in delayed feedback for the development team.
2. **Limitations of existing techniques**:
- **Test suite reduction**: It accelerates regression testing by reducing the number of test cases, but some important test cases may be lost.
- **Test case selection**: It selects test cases that cover the modified code, but may omit other important test cases.
- **Test case prioritization**: By re - arranging the execution order of test cases to detect more defects earlier, but existing methods are mainly based on code coverage and ignore other useful information sources.
3. **Fault - proneness estimation**:
- Fault - proneness estimation can be obtained from the defect history of the project through defect prediction methods and is used to estimate the failure probability of code units.
- However, simply using fault - proneness to prioritize test cases may lead to similar test cases being given too high a priority, thus affecting the diversity of testing.
4. **Test case diversity**:
- Intuitively, test cases with similar characteristics may have similar defect - detecting capabilities.
- Therefore, considering the diversity of test cases can improve the effectiveness of testing.
### The proposed method
This paper proposes a new method that combines fault - proneness estimation and test case diversity. The specific steps are as follows:
1. **Defect prediction**:
- Use defect prediction methods to estimate the fault - proneness of each code unit.
- The defect prediction method is modeled based on the historical defect data and code features of the project.
2. **Test case clustering**:
- Use the hierarchical clustering algorithm to group test cases, with each group containing similar test cases.
3. **Internal prioritization**:
- Within each cluster, use the code - coverage - based method to prioritize test cases.
4. **Comprehensive prioritization**:
- Comprehensively rank the test cases in each cluster according to fault - proneness estimation and traditional TCP methods to generate the final test case execution order.
### Experiment and evaluation
- **Experimental data**: Experiments were carried out in 5 real - world open - source projects, with a total of 357 versions.
- **Evaluation metrics**: Use the percentage of First - Fail Metric and the Average Percentage of Fault Detection (APFD) to evaluate the performance of different methods.
### Main contributions
1. **A new method combining fault - proneness and diversity**:
- Propose a new TCP method that combines fault - proneness estimation and test case diversity, improving the efficiency of defect detection.
2. **A customized defect prediction method**:
- Design a defect prediction method suitable for regression testing scenarios, which can work effectively with only a small number of recorded defects.
3. **Empirical research**:
- Conducted extensive experiments in multiple real projects to verify the effectiveness of the proposed method.
### Conclusion
The method proposed in this paper performs better than the traditional code - coverage - based TCP method in empirical research, proving that combining fault - proneness estimation and test case diversity can significantly improve the efficiency and effectiveness of regression testing.