Automated testing of metamodels and code co-evolution
Kebaili, Zohra Kaouter,Khelladi, Djamel Eddine,Acher, Mathieu,Barais, Olivier
DOI: https://doi.org/10.1007/s10270-024-01245-2
2024-12-10
Software & Systems Modeling
Abstract:Metamodels are cornerstone in MDE. They define the different domain concepts and the relations between them. A metamodel is also used to generate concrete artifacts such as code. Developers then rely on the generated code to build their language services and tooling, e.g., editors, checkers. To check the behavior of their client code, developers write or generate unit tests. As metamodels evolve between releases, the generated code is automatically updated. As a consequence, the additional developers' code is impacted and is co-evolved accordingly for each release. However, there is no guarantee that the co-evolution of the code is performed correctly. One way to do so is to rerun all the tests after each code co-evolution, which is expensive and time-consuming. This paper proposes an automatic solution for tracing impacted tests due to metamodel evolution. Thus, we end up matching metamodel changes with impacted code methods and their corresponding tests in both the original and evolved versions of a given project. After that, we map the two versions of the impacted tests and compare them to analyze the behavior of the code before and after its evolution due to the metamodel evolution. In particular, we implemented an Eclipse plug-in that allows tracing, mapping, execution, and reporting back the results to the developers for easier in-depth analysis of the effect of metamodel evolutions rather than analyzing the whole test suite. We first ran a user study experiment to gain evidence on the difficulty or not of the manual task of tracing impacted tests. We found that manually tracing the tests impacted by the evolution of the metamodel is a hard and error-prone task. Not only the participants could not trace all tests, but they even wrongly traced non-impacted tests. We then evaluated our approach on 18 Eclipse projects from OCL, Modisco, Papyrus, and EMF over several evolved versions of metamodels. For the 14 projects without manual tests, we generated a test suite for each release with the state-of-the-art tool EvoSuite. The results show that we successfully traced the impacted tests automatically by selecting 1608 out of 34,612 tests due to 473 metamodel changes. When running the traced tests before and after co-evolution, we observed cases indicating possibly both behaviorally correct and incorrect code co-evolution. Finally, we reached gains representing, on average, a reduction of 88% in the number of tests and 84% in the execution time.
computer science, software engineering