A Catalog of Transformations to Remove Smells From Natural Language Tests

Manoel Aranda,Naelson Oliveira,Elvys Soares,Márcio Ribeiro,Davi Romão,Ullyanne Patriota,Rohit Gheyi,Emerson Souza,Ivan Machado

DOI: https://doi.org/10.1145/3661167.3661225

2024-04-26

Abstract:Test smells can pose difficulties during testing activities, such as poor maintainability, non-deterministic behavior, and incomplete verification. Existing research has extensively addressed test smells in automated software tests but little attention has been given to smells in natural language tests. While some research has identified and catalogued such smells, there is a lack of systematic approaches for their removal. Consequently, there is also a lack of tools to automatically identify and remove natural language test smells. This paper introduces a catalog of transformations designed to remove seven natural language test smells and a companion tool implemented using Natural Language Processing (NLP) techniques. Our work aims to enhance the quality and reliability of natural language tests during software development. The research employs a two-fold empirical strategy to evaluate its contributions. First, a survey involving 15 software testing professionals assesses the acceptance and usefulness of the catalog's transformations. Second, an empirical study evaluates our tool to remove natural language test smells by analyzing a sample of real-practice tests from the Ubuntu OS. The results indicate that software testing professionals find the transformations valuable. Additionally, the automated tool demonstrates a good level of precision, as evidenced by a F-Measure rate of 83.70%

Software Engineering

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the problem of "Test Smells" in Natural Language Tests that exist but have not received sufficient attention. Specifically:

1. **Insufficient existing research**: Although a large number of studies have explored test smells in automated software testing and provided methods for identifying and removing these smells, there is still relatively little research on smells in natural language tests.

2. **Lack of systematic methods**: Currently, there is a lack of systematic methods for removing smells in natural language tests, and also a lack of automated tools to identify and handle these problems.

3. **Improve test quality**: In order to improve the quality and reliability of natural language tests, the author proposes a set of transformation rules and develops a tool based on Natural Language Processing (NLP) technology to automatically identify and remove seven specific natural language test smells.

### Specific goals of the paper

- **Construct a transformation catalogue**: Create a catalogue of transformation rules for seven natural language test smells to help testers better understand and handle these smells.

- **Develop an automated tool**: Develop an automated tool that can automatically identify and remove smells in natural language tests according to the above - mentioned transformation rules.

- **Evaluate effectiveness**: Evaluate the effectiveness of the proposed transformation rules and tools through two empirical studies:
- A survey of 15 software testing professionals to evaluate their views on the transformation rules.
- An empirical study using actual natural language test samples of the Ubuntu operating system to evaluate the accuracy of the tool.

### Test smells involved

The seven natural language test smells mentioned in the paper include:

- **Unverified Action**: There is no verification of the result of an action in the test step.
- **Misplaced Precondition**: The precondition is wrongly placed in the test step.
- **Misplaced Action**: The action appears in the verification list.
- **Misplaced Verification**: The verification appears in the action list.
- **Eager Action**: Multiple actions are included in one step.
- **Ambiguous Test**: The test description is ambiguous and easily misunderstood.
- **Conditional Test**: The test step contains conditional statements, resulting in uncertainty and variability.

By solving these problems, the author hopes to improve the quality and reliability of natural language tests, thereby improving the testing activities in the software development process.