Towards Comprehensive Evaluation for Android Automated Testing Tools
ZHONG Yi,SHI Meng-Yu,FANG Chun-Rong,ZHAO Zhi-Hong,CHEN Zhen-Yu
DOI: https://doi.org/10.13328/j.cnki.jos.006701
2023-01-01
Journal of Software
Abstract:Automated testing tools are the primary means of quality assurance for Android applications. With the increase in Android version diversity, underlying hardware variability（fragmentation）, and logical complexity, automated testing faces new challenges.Numerous automated testing tools have been developed in recent years to address the above issues. However, there are vast tools with various testing focuses, making it hard for testers to choose the right one. To help testers select the best tool for testing and achieve a unified evaluation for automated testing tools, a multi-characteristic comprehensive evaluation of the Android automated testing（CEAT）method is proposed and an easy-to-use platform is implemented for testers. CEAT introduces three widely accepted evaluation metrics:code coverage, exception detection rate, fusion multi-version compatibility score, and further introduces mutation kill rate based on the mutation testing concept, and UI control widget coverage from the perspective of the user. The five metrics constitute the whole CEAT system, thus realizing a comprehensive multi-dimensional evaluation of Android automated testing tool. To verify the effectiveness of CEAT, a set of 1,089 mutated applications is generated for testing, the experiments are deployed in a real-world cluster containing six mobile devices, and 5,040 test tasks are executed for the testing tools. The results suggest that:（i） the five indicators evaluate the automated testing tools from different perspectives, reflecting the testing performance of different tools in a more multi-dimensional way and validating the effectiveness of CEAT;（ii） CEAT supports testers to assign different weights to the five metrics and obtain comprehensive evaluation results depending on the practical testing requirements, which has certain flexibility;（iii） CEAT automatically reconstructs the APP to obtain mutant APPs and set a specific platform for testing the tool, making it convenient to operate. CEAT effectively provides a reference for testers to select the best Android automated testing tool according to different testing requirements.