Identifying Similar Test Cases That Are Specified in Natural Language

Markos Viggiato,Dale Paas,Chris Buzon,Cor-Paul Bezemer

DOI: https://doi.org/10.48550/arXiv.2110.07733

2021-10-15

Abstract:Software testing is still a manual process in many industries, despite the recent improvements in automated testing techniques. As a result, test cases are often specified in natural language by different employees and many redundant test cases might exist in the test suite. This increases the (already high) cost of test execution. Manually identifying similar test cases is a time-consuming and error-prone task. Therefore, in this paper, we propose an unsupervised approach to identify similar test cases. Our approach uses a combination of text embedding, text similarity and clustering techniques to identify similar test cases. We evaluate five different text embedding techniques, two text similarity metrics, and two clustering techniques to cluster similar test steps and four techniques to identify similar test cases from the test step clusters. Through an evaluation in an industrial setting, we showed that our approach achieves a high performance to cluster test steps (an F-score of 87.39%) and identify similar test cases (an F-score of 83.47%). Furthermore, a validation with developers indicates several different practical usages of our approach (such as identifying redundant and legacy test cases), which help to reduce the testing manual effort and time.

Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the problem of manually identifying similar or duplicate test cases in software testing. Although automated testing techniques have improved a great deal, in many industries, software testing is still a manual process. This results in test cases usually being written in natural language by different employees, and there may be a large number of redundant test cases in the test suite. Manually identifying these similar test cases is both time - consuming and error - prone, increasing the cost of test execution. Therefore, the paper proposes an unsupervised method to automatically identify similar test cases in order to reduce test redundancy and lower the workload and time cost of manual testing.

Specifically, the paper solves this problem through the following steps:

1. **Pre - process test cases**: Pre - process the original test cases, including word segmentation, spell - checking, removing stop words, lemmatization, etc., to ensure the quality of the data.
2. **Cluster test steps**: Use text embedding, text similarity and clustering techniques to cluster test steps. The paper evaluates five different text embedding techniques (Word2Vec, BERT, Sentence - BERT, Universal Sentence Encoder and TF - IDF), two text similarity metrics (Word Mover's Distance and cosine similarity) and two clustering techniques (hierarchical clustering and K - means clustering) to find the most effective combination.
3. **Identify test case similarity**: Based on the test step clusters generated in the second stage, combined with the test case names, identify similar test cases. The paper compares four different techniques to calculate the similarity scores between test cases (simple overlap, Jaccard similarity and cosine similarity) to determine the similarity of test cases.

Through this method, the paper aims to provide an efficient, automated solution to help quality assurance engineers and developers identify and remove redundant and obsolete test cases, thereby improving test efficiency and quality.