Abstract:To ensure app compatibility and smoothness of user experience across diverse devices and platforms, developers have to perform cross-device, cross-platform testing of their apps, which is laborious. There comes a recently increasing trend of using a record and replay approach to facilitate the testing process. However, the graphic user interface (GUI) of an app running on different devices and platforms differs dramatically. This complicates the record and replay process as the presence, appearance and layout of the GUI widgets in the recording phase and replaying phase can be inconsistent. Existing techniques resort to instrumenting into the underlying system to obtain the app metadata for widget identification and matching between various devices. But such intrusive practices are limited by the accessibility and accuracy of the metadata on different platforms. On the other hand, several recent works attempt to derive the GUI information by analyzing the GUI image. Nevertheless, their performance is curbed by the applied preliminary visual approaches and the failure to consider the divergence of the same GUI displayed on different devices. To address the challenge, we propose a non-intrusive cross-device and cross-platform system NiCro. NiCro utilizes the state-of-the-art GUI widget detector to detect widgets from GUI images and then analyses a set of comprehensive information to match the widgets across diverse devices. At the system level, NiCro can interact with a virtual device farm and a robotic arm system to perform cross-device, cross-platform testing non-intrusively. We first evaluated NiCro by comparing its multi-modal widget and GUI matching approach with 4 commonly used matching techniques. Then, we further examined its overall performance on 8 various devices, using it to record and replay 107 test cases of 28 popular apps and the home page to show its effectiveness.

Deep Learning-Based Mobile Application Isomorphic GUI Identification for Automated Robotic Testing

Semantic Similarity-Based Mobile Application Isomorphic Graphical User Interface Identification

Construction of Semantic Model for GUI of Mobile Applications Using Deep Learning.

Visual Identification of Mobile App GUI Elements for Automated Robotic Testing.

Visual Judgment Approach of Isomorphic GUI for Automated Mobile App Testing

DroidGamer: Android Game Testing with Operable Widget Recognition by Deep Learning*

Humanoid: A Deep Learning-Based Approach to Automated Black-box Android App Testing.

A reinforcement learning-based approach to testing GUI of moblie applications

Automatic Construction of Graphical User Interfaces Semantic Models Using Robots for Mobile Application Testings

A Novel Automated GUI Testing Echnology Based on Image Recognition

Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcement Learning

Humanoid: A Deep Learning-based Approach to Automated Black-box Android App Testing

DNN-Based Image Classification for Software GUI Testing.

Practical Non-Intrusive GUI Exploration Testing with Visual-based Robotic Arms

Deeply Reinforcing Android GUI Testing with Deep Reinforcement Learning.

Learning-Replay Based Automated Robotic Testing for Mobile App

YOLOv5-MGC: GUI Element Identification for Mobile Applications Based on Improved YOLOv5

NiCro: Purely Vision-based, Non-intrusive Cross-Device and Cross-Platform GUI Testing

Facilitating Monkey Test by Detecting Operable Regions in Rendered GUI of Mobile Game Apps

Adaptive attention fusion network for cross-device GUI element re-identification in crowdsourced testing

Software GUI automatic testing systems based on the cloud platform and deep learning