Mobile Application Testing: A Tutorial

Jerry Gao,Xiaoying Bai,Wei-Tek Tsai,Tadahiro Uehara

DOI: https://doi.org/10.1109/mc.2013.445

2014-01-01

Computer

Abstract:To cope with frequent upgrades of mobile devices and technologies, engineers need a reusable and cost-effective environment for testing mobile applications and an elastic infrastructure to support large-scale test automation.

What problem does this paper attempt to address?

Towards Scalable Automated Mobile App Testing

Ranveer Chandra,Börje F. Karlsson,Nicholas D. Lane,Mike Chieh-Jan Liang,Suman Nath,Jitendra Padhye,Lenin Ravindranath Sivalingam,Feng Zhao

2014-01-01

Abstract:The mobile app ecosystem continues to rapidly grow in importance as an increasing proportion of our daily computing needs shift away from desktop machines in favor of mobile devices. However, mobile apps must cope with extremely diverse operating conditions due to factors like device fragmentation, wireless network heterogeneity and varied user behavior. App developers and distributors (i.e., operators of app marketplaces) lack testing tools that can effectively account for such diversity and, as a result, app failures and performance bugs (e.g., excessive energy consumption) are commonly found in mobile apps today. Towards addressing this challenge to mobile app development, we have developed key techniques for scalable automated mobile app testing within two prototype app testing services – VanarSena and Caiipa. In this paper, we describe SMASH – our vision for a single unified cloud-based mobile app testing service that will combine the strengths of the VanarSena and Caiipa systems towards solving the complexities presently faced by testers of mobile apps.
Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing

Mario Linares Vasquez,Kevin Moran,Denys Poshyvanyk

DOI: https://doi.org/10.48550/arXiv.1801.06267

2018-01-19

Abstract:Mobile app development involves a unique set of challenges including device fragmentation and rapidly evolving platforms, making testing a difficult task. The design space for a comprehensive mobile testing strategy includes features, inputs, potential contextual app states, and large combinations of devices and underlying platforms. Therefore, automated testing is an essential activity of the development process. However, current state of the art of automated testing tools for mobile apps poses limitations that has driven a preference for manual testing in practice. As of today, there is no comprehensive automated solution for mobile testing that overcomes fundamental issues such as automated oracles, history awareness in test cases, or automated evolution of test cases. In this perspective paper we survey the current state of the art in terms of the frameworks, tools, and services available to developers to aid in mobile testing, highlighting present shortcomings. Next, we provide commentary on current key challenges that restrict the possibility of a comprehensive, effective, and practical automated testing solution. Finally, we offer our vision of a comprehensive mobile app testing framework, complete with research agenda, that is succinctly summarized along three principles: Continuous, Evolutionary and Large-scale (CEL).

Software Engineering
Automated Visual Testing for Mobile Apps in an Industrial Setting

Dezhi Ran,Zongyang Li,Chenxu Liu,Wenyu Wang,Weizhi Meng,Xionglin Wu,Hui Jin,Jing Cui,Xing Tang,Tao Xie

DOI: https://doi.org/10.1145/3510457.3513027

2022-01-01

Abstract:User Interface (UI) testing has become a common practice for quality assurance of industrial mobile applications (in short as apps). While many automated tools have been developed, they often do not satisfy two major industrial requirements that make a tool desirable in industrial settings: high applicability across platforms (e.g., Android, iOS, AliOS, and Harmony OS) and high capability to handle apps with non-standard UI elements (whose internal structures cannot be acquired using platform APIs). Toward addressing these industrial requirements, automated visual testing emerges to take only device screenshots as input in order to support automated test generation. In this paper, we report our experiences of developing and deploying VTest, our industrial visual testing framework to assure high quality of Taobao, a highly popular industrial app with about one billion monthly active users. VTest includes carefully designed techniques and infrastructure support, outperforming Monkey (which has been popularly deployed in industry and shown to perform superiorly or similarly compared to state-of-the-art tools) with 87.6% more activity coverage. VTEST has been deployed both internally in Alibaba and externally in the Software Green Alliance to provide testing services for top smart-phone vendors and app vendors in China. We summarize five major lessons learned from developing and deploying VTEST.
A Test Automation Solution On Gui Functional Test

zhu xiaochun,zhou bo,li juefeng,gao qiu

DOI: https://doi.org/10.1109/INDIN.2008.4618325

2008-01-01

Abstract:Although it is widely believed that software quality will be improved by the use of automated testing, automation Is still not well-off in industry today. There are quite a few issues of traditional software test automation mode, for example, high cost, knowledge barriers, and management troubles.In the paper, a test automation solution on GUI functional test is proposed. The solution integrates test case generation and selection, test case execution, and test reporting to facilitate testing. It introduces the concept of test driver which is designed to take over the communication between test cases and the execution engine. At last, some enhancements are proposed for future work.
Mobile Testing-as-a-Service (MTaaS) -- Infrastructures, Issues, Solutions and Needs

Jerry Gao,Wei-Tek Tsai,Ray Paul,Xiaoying Bai,Tadahiro Uehara

DOI: https://doi.org/10.1109/HASE.2014.30

2014-01-01

Abstract:With the rapid advance of mobile computing technology and wireless networking, there is a significant increase of mobile subscriptions. This drives a strong demand for development and validation of mobile APPs and SaaS applications on mobile web. This paper is written to offer informative and insightful discussion about mobile testing-as-a-service (MTaaS), including its basic concepts, motivations, distinct features and requirements, test environments, and different approaches. Moreover, it presents a test process in MTaaS and three different approaches. Furthermore, the paper proposes one mobile test cloud infrastructure for mobile TaaS, and discusses the required mobile test frameworks and environments. Finally, the paper addresses existing issues, challenges, and emergent needs.
To the Attention of Mobile Software Developers: Guess What, Test your App!

Luis Cruz,Rui Abreu,David Lo

DOI: https://doi.org/10.48550/arXiv.1902.02610

2019-02-15

Abstract:Software testing is an important phase in the software development life-cycle because it helps in identifying bugs in a software system before it is shipped into the hand of its end users. There are numerous studies on how developers test general-purpose software applications. The idiosyncrasies of mobile software applications, however, set mobile apps apart from general-purpose systems (e.g., desktop, stand-alone applications, web services). This paper investigates working habits and challenges of mobile software developers with respect to testing. A key finding of our exhaustive study, using 1000 Android apps, demonstrates that mobile apps are still tested in a very ad hoc way, if tested at all. However, we show that, as in other types of software, testing increases the quality of apps (demonstrated in user ratings and number of code issues). Furthermore, we find evidence that tests are essential when it comes to engaging the community to contribute to mobile open source software. We discuss reasons and potential directions to address our findings. Yet another relevant finding of our study is that Continuous Integration and Continuous Deployment (CI/CD) pipelines are rare in the mobile apps world (only 26% of the apps are developed in projects employing CI/CD) --- we argue that one of the main reasons is due to the lack of exhaustive and automatic testing.

Software Engineering
User Guided Automation for Testing Mobile Apps

Xiujiang Li,Yanyan Jiang,Yepang Liu,Chang Xu,Xiaoxing Ma,Jian Lu

DOI: https://doi.org/10.1109/apsec.2014.13

2014-01-01

Abstract:Mobile devices are gradually taking over traditional computers' dominance in human lives. With the ever-increasing shipment of mobile apps running on these devices, their quality issues become a severe challenge. Although automated testing techniques are being widely studied, they mostly fall short of handling mobile apps' complex interactions, e.g., A finger swipe or device shaking gesture, leading to inadequate testing. In this paper, we present a novel User Guided Automation (UGA) technique to address testing challenges incurred by such complex interactions. UGA exploits user insights to complement automated testing techniques by recording user-guided app executions, replaying apps to certain stop points, and systematically exploring state space from these stop points. We implemented our work as a prototype UGA tool on Android platform and evaluated it on seven real-world Android apps. Evaluation results show that UGA achieved 1.59-21.78× improvement in terms of method coverage over state-of-the-art automated techniques in testing mobile apps.
A reinforcement learning-based approach to testing GUI of moblie applications

Chuanqi Tao,Fengyu Wang,Yuemeng Gao,Hongjing Guo,Jerry Gao

DOI: https://doi.org/10.1007/s11280-024-01252-9

2024-02-23

World Wide Web

Abstract:With the popularity of mobile devices, the software market of mobile applications has been booming in recent years. Android applications occupy a vast market share. However, the applications inevitably contain defects. Defects may affect the user experience and even cause severe economic losses. This paper proposes ATAC and ATPPO, which apply reinforcement learning to Android GUI testing to mitigate the state explosion problem. The article designs a new reward function and a new state representation. It also constructs two GUI testing models (ATAC and ATPPO) based on A2C and PPO algorithms to save memory space and accelerate training speed. Empirical studies on twenty open-source applications from GitHub demonstrate that: (1) ATAC performs best in 16 of 20 apps in code coverage and defects more exceptions; (2) ATPPO can get higher code coverage in 15 of 20 apps and defects more exceptions; (3) Compared with state-of-art tools Monkey and ARES, ATAC, and ATPPO shows higher code coverage and detects more errors. ATAC and ATPPO can not only cover more code coverage but also can effectively detect more exceptions. This paper also introduces Finite-State Machine into the reinforcement learning framework to avoid falling into the local optimal state, which provides high-level guidance for further improving the test efficiency.

computer science, information systems, software engineering
An Automated Testing Platform for Mobile Applications

Xin Ma,Ning Wang,Peizhang Xie,Jungui Zhou,Xiaofang Zhang,Chunrong Fang

DOI: https://doi.org/10.1109/QRS-C.2016.25

2016-01-01

Abstract:With the growing popularity and complexity of mobile apps, quality assurance becomes more and more important in mobile app development. Unfortunately, Android is also suffering from the notorious fragmentation problem. To tackle these problems, we introduce an automated testing platform, BugRocket. BugRocket combines a distributed testing system with automated testing techniques equipped on mobile devices. In this paper, we developed a toolset for automated testing, and set up such a testing platform with 40 of the most popular Android devices. An case study is conducted and the results show that BugRocket can work for functional testing and compatibility testing. Besides, BugRocket can record a failed run as long as annotated GUI model and system logs to alleviate locating bugs and bug fixing.
Take the Blue Pill: Pursuing Mobile App Testing Fidelity, Efficiency, and Accessibility with Virtual Device Farms

Hao Lin,Jiaxing Qiu,Hongyi Wang,Zhenhua Li,Liangyi Gong,Di Gao,Yunhao Liu,Feng Qian,Zhao Zhang,Ping Yang,Tianyin Xu

DOI: https://doi.org/10.1145/3665112.3665114

2024-01-01

Abstract:For more than a decade, researchers have been extensively exploring mobile app testing on virtual devices [1-9], which are software-emulated mobile devices running on commodity servers, in a similar vein as virtual machines (VM) in the cloud. Building on server virtualization, virtual devices naturally inherit the benefits of VM, such as scalability, elasticity, and cost efficiency. Moreover, virtualization enables useful features not offered by physical devices, such as service instrumentation [3], whole-system snapshot [9], and memory introspection [1,2], atop which a series of advanced testing and debugging techniques are developed.
GUI-Guided Test Script Repair for Mobile Apps

Minxue Pan,Tongtong Xu,Yu Pei,Zhong Li,Tian Zhang,Xuandong Li

DOI: https://doi.org/10.1109/TSE.2020.3007664

IF: 7.4

2022-01-01

IEEE Transactions on Software Engineering

Abstract:Graphical User Interface (GUI) testing is widely used to test mobile apps. As mobile apps are frequently updated and need repeated testing, to reduce the test cost, their test cases are often coded as scripts to enable automated execution using test harnesses/tools. When those mobile apps evolve, many of the test scripts, however, may become broken due to changes made to the app GUIs. While it is ...
ATOM: Automatic Maintenance of GUI Test Scripts for Evolving Mobile Applications

Xiao Li,Nana Chang,Yan Wang,Haohua Huang,Yu Pei,Linzhang Wang,Xuandong Li

DOI: https://doi.org/10.1109/icst.2017.22

2017-01-01

Abstract:The importance of regression testing in assuring the integrity of a program after changes is well recognized. One major obstacle in practicing regression testing is in maintaining tests that become obsolete due to evolved program behavior or specification. For mobile apps, the problem of maintaining obsolete GUI test scripts for regression testing is even more pressing. Mobile apps rely heavily on the correct functioning of their GUIs to compete on the market and provide good user experiences. But on the one hand, GUI tests break easily when changes happen to the GUI, On the other hand, mobile app developers often need to fight for a tight feedback loop and are left with limited time for test maintenance. In this paper, we propose a novel approach, called ATOM, to automatically maintain GUI test scripts of mobile apps for regression testing. ATOM uses an event sequence model to abstract possible event sequences on a GUI and a delta ESM to abstract the changes made to the GUI. Given both models as input, ATOM automatically updates the test scripts written for a base version app to reflect the changes. In an experiment with 22 versions from 11 production Android apps, ATOM updated all the test scripts affected by the version change, the updated scripts achieve over 80% of the coverage by the original scripts on the base version app, all except one set of updated scripts preserve over 60% of the actions in the original test scripts.
Mobile-D: An Agile Approach for Mobile Application Development

Pekka Abrahamsson,Antti Hanhineva,Hanna Hulkko,Tuomas Ihme,Juho Jäälinoja,Mikko Korkala,Juha Koskela,Pekka Kyllönen,Outi Salo

DOI: https://doi.org/10.48550/arXiv.1709.06820

2017-09-20

Software Engineering

Abstract:Mobile phones have been closed environments until recent years. The change brought by open platform technologies such as the Symbian operating system and Java technologies has opened up a significant business opportunity for anyone to develop application software such as games for mobile terminals. However, developing mobile applications is currently a challenging task due to the specific demands and technical constraints of mobile development. Furthermore, at the moment very little is known about the suitability of the different development processes for mobile application development. Due to these issues, we have developed an agile development approach called Mobile-D. The Mobile-D approach is briefly outlined here and the experiences gained from four case studies are discussed.
Facilitating Reusable and Scalable Automated Testing and Analysis for Android Apps.

Zhanshuai Meng,Yanyan Jiang,Chang Xu

DOI: https://doi.org/10.1145/2875913.2875937

2015-01-01

Abstract:Mobile apps are prevalent in everyone's daily life. However, apps are oftentimes defective, undermining their convenience, and therefore automated testing and analysis of apps are developed to enhance apps' quality. As these advanced technologies become increasingly effective and complicated, prototyping such a tool also becomes a challenge. To facilitate easy development of high-quality testing and analysis tools that work with real-world apps, we in this paper present the design and implementation of ATT (Android Testing Toolkit), for crafting reusable and scalable testing and analysis on Android apps. ATT consists of integrated tools and APIs for event generation, profiling and program instrumentation, and can be distributed over cloud platforms for scalable testing and analysis. We qualitatively evaluated the applicability of ATT by demonstrating implementation of a series of existing and enhanced work (RERAN, Monkey+ and UGA) upon ATT, and quantitatively studied the scalability of parallelized UGA on a cloud platform with five real-world apps. We believe that our ATT tool would facilitate development of more advanced testing and analysis approaches for Android apps.
How do Developers Test Android Applications?

Mario Linares Vasquez,Carlos Bernal-Cardenas,Kevin Moran,Denys Poshyvanyk

DOI: https://doi.org/10.48550/arXiv.1801.06268

2018-01-19

Abstract:Enabling fully automated testing of mobile applications has recently become an important topic of study for both researchers and practitioners. A plethora of tools and approaches have been proposed to aid mobile developers both by augmenting manual testing practices and by automating various parts of the testing process. However, current approaches for automated testing fall short in convincing developers about their benefits, leading to a majority of mobile testing being performed manually. With the goal of helping researchers and practitioners - who design approaches supporting mobile testing - to understand developer's needs, we analyzed survey responses from 102 open source contributors to Android projects about their practices when performing testing. The survey focused on questions regarding practices and preferences of developers/testers in-the-wild for (i) designing and generating test cases, (ii) automated testing practices, and (iii) perceptions of quality metrics such as code coverage for determining test quality. Analyzing the information gleaned from this survey, we compile a body of knowledge to help guide researchers and professionals toward tailoring new automated testing approaches to the need of a diverse set of open source developers.

Software Engineering
Human-Machine Collaborative Testing for Android Applications

Yuying Li,Yang Feng,Rui Hao,Zhenyu Chen

DOI: https://doi.org/10.1109/qrs60937.2023.00050

2023-01-01

Abstract:Android applications bring many challenges for testing due to the notorious fragmentation issues and their diverse usage environments. Even though classic crowdsourced testing can improve the usability and reliability of Android applications, it often requires many efforts and resources. Meanwhile, on the other hand, automated analysis techniques provide highly efficient testing solutions for Android applications, which can use rich test resources and computing power to save manual test costs. Therefore, to leverage the advantages of both manual and automated testing, in this paper, we propose an approach that combines classic static program analysis and crowdsourced testing to implement human-machine collaborative testing for Android applications. We first employ the static analysis technique to model the possible GUI window sequences into window transition graphs (WTG). Then, we use a depth-first search algorithm to traverse the WTG and generate the testing task lists. In the testing process, we recommend these tasks for testers and adjust the task priority based on user feedback to optimize collaborative testing efficiency and effectiveness. To validate our approach, we experiment it with 9 open-source Android applications. The experimental results show that the path coverage rate of human-machine collaborative testing is 18.2% higher than that of classic crowdsourced testing and can reduce duplicate bug reports by 17.0 percent and improve testing efficiency.
Towards the quality improvement of cross-platform mobile applications

Matias Martinez,Sylvain Lecomte

DOI: https://doi.org/10.48550/arXiv.1701.06767

2017-03-02

Abstract:During last ten years, the number of smartphones and mobile applications has been constantly growing. Android, iOS and Windows Mobile are three mobile platforms that cover almost all smartphones in the world in 2017. Developing a mobile app involves first to choose the platforms the app will run, and then to develop specific solutions (i.e., native apps) for each chosen platform using platform-related toolkits such as AndroidSDK. Across-platform mobile application is an app that runs on two or more mobile platforms. Several frameworks have been proposed to simplify the development of cross-platform mobile applications and to reduce development and maintenance <a class="link-external link-http" href="http://costs.They" rel="external noopener nofollow">this http URL</a> are called cross-platform mobile app development <a class="link-external link-http" href="http://frameworks.However" rel="external noopener nofollow">this http URL</a>, to our knowledge, the life-cycle and the quality of cross-platforms mobile applications built using those frameworks have not been studied in depth. Our main goal is to first study the processes of development and maintenance of mobile applications built using cross-platform mobile app development frameworks, focusing particularly on the bug-fixing activity. Then, we aim at defining tools for automated repairing bugs from cross-platform mobile applications.

Software Engineering,Programming Languages
Practical, Automated Scenario-based Mobile App Testing

Shengcheng Yu,Chunrong Fang,Mingzhe Du,Zimin Ding,Zhenyu Chen,Zhendong Su

DOI: https://doi.org/10.1109/TSE.2024.3414672

2024-06-12

Abstract:The importance of mobile application (app) quality insurance is increasing with the rapid development of the mobile Internet. Automated test generation approaches, as a dominant direction of app quality insurance, follow specific models or strategies, targeting at optimizing the code coverage. Such approaches lead to a huge gap between testing execution and app business logic. Test scripts developed by human testers consider business logic by focusing on testing scenarios. Due to the GUI-intensive feature of mobile apps, human testers always understand app GUI to organize test scripts for scenarios. This inspires us to utilize domain knowledge from app GUI understanding for scenario-based test generation. In this paper, we propose a novel approach, ScenTest, for scenario-based mobile app testing with event knowledge graph (EKG) via GUI image understanding. ScenTest tries to start automated testing by imitating human practices and integrating domain knowledge into scenario-based mobile app testing, realizing fully automated testing on target testing scenarios for the first time. ScenTest extracts four kinds of entities and five kinds of corresponding relationships from crowdsourced test reports, where the test events and app GUI information are presented, and constructs the EKGs for specific scenarios. Then, ScenTest conducts test generation for specific scenarios on different apps with the guidance of EKG with the combination consideration of app current state and testing context. We conduct an evaluation on ScenTest on different aspects. The results show that the test generation of ScenTest on the basis of EKG is effective, and ScenTest can reveal 80+ distinct real-world bugs in specific scenarios compared with representative baselines.

Software Engineering
Practical GUI Testing of Android Applications Via Model Abstraction and Refinement

Tianxiao Gu,Chengnian Sun,Xiaoxing Ma,Chun Cao,Chang Xu,Yuan Yao,Qirun Zhang,Jian Lu,Zhendong Su

DOI: https://doi.org/10.1109/icse.2019.00042

2019-01-01

Abstract:This paper introduces a new, fully automated model-based approach for effective testing of Android apps. Different from existing model-based approaches that guide testing with a static GUI model ( i.e. , the model does not evolve its abstraction during testing, and is thus often imprecise), our approach dynamically optimizes the model by leveraging the runtime information during testing. This capability of model evolution significantly improves model precision, and thus dramatically enhances the testing effectiveness compared to existing approaches, which our evaluation confirms. We have realized our technique in a practical tool, Ape. On 15 large, widely-used apps from the Google Play Store, Ape outperforms the state-of-the-art Android GUI testing tools in terms of both testing coverage and the number of detected unique crashes. To further demonstrate Ape's effectiveness and usability, we conduct another evaluation of Ape on 1,316 popular apps, where it found 537 unique crashes. Out of the 38 reported crashes, 13 have been fixed and 5 have been confirmed.
Cloud testing-issues, challenges, needs and practice

Jerry Gao,Xiaoying Bai,Wei-Tek Tsai

2011-01-01

Abstract:Cloud computing not only changes the way of obtaining computing resources (such as computers, infrastructures, data storage, and application services), but also changes the way of managing and delivering computing services, technologies, and solutions. Cloud computing leads an opportunity in offering testing as a service (TaaS) for SaaS and clouds. Meanwhile, it causes new issues, challenges and needs in software testing, particular in testing clouds and cloud-based applications. This paper provides a comprehensive tutorial on cloud testing and cloud-based application testing. It answers the common questions raised by engineers and managers, and it provides clear concepts, discusses the special objectives, features, requirements, and needs in cloud testing. It offers a clear comparative view between web-based software testing and cloud-based application testing. In addition, it examines the major issues, challenges, and needs in testing cloud-based software applications. Furthermore, it also summarizes and compares different commercial products and solutions supporting cloud testing as services.

Mobile Application Testing: A Tutorial

Towards Scalable Automated Mobile App Testing

Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing

Automated Visual Testing for Mobile Apps in an Industrial Setting

A Test Automation Solution On Gui Functional Test

Mobile Testing-as-a-Service (MTaaS) -- Infrastructures, Issues, Solutions and Needs

To the Attention of Mobile Software Developers: Guess What, Test your App!

User Guided Automation for Testing Mobile Apps

A reinforcement learning-based approach to testing GUI of moblie applications

An Automated Testing Platform for Mobile Applications

Take the Blue Pill: Pursuing Mobile App Testing Fidelity, Efficiency, and Accessibility with Virtual Device Farms

GUI-Guided Test Script Repair for Mobile Apps

ATOM: Automatic Maintenance of GUI Test Scripts for Evolving Mobile Applications

Mobile-D: An Agile Approach for Mobile Application Development

Facilitating Reusable and Scalable Automated Testing and Analysis for Android Apps.

How do Developers Test Android Applications?

Human-Machine Collaborative Testing for Android Applications

Towards the quality improvement of cross-platform mobile applications

Practical, Automated Scenario-based Mobile App Testing

Practical GUI Testing of Android Applications Via Model Abstraction and Refinement

Cloud testing-issues, challenges, needs and practice