Abstract:Software applications have been playing an increasingly important role in various aspects of society. In particular, mobile apps and web apps are the most prevalent among all applications and are widely used in various industries as well as in people's daily lives. To help ensure mobile and web app quality, many approaches have been introduced to improve app GUI testing via automated exploration. Despite the extensive effort, existing approaches are still limited in reaching high code coverage, constructing high-quality models, and being generally applicable. Reinforcement learning-based approaches are faced with difficult challenges, including effective app state abstraction, reward function design, etc. Moreover, they heavily depend on the specific execution platforms, thus leading to poor generalizability and being unable to adapt to different platforms. We propose PIRLTest, an effective platform-independent approach for app testing. It utilizes computer vision and reinforcement learning techniques in a novel, synergistic manner for automated testing. It extracts the GUI widgets from GUI pages and characterizes the corresponding GUI layouts, embedding the GUI pages as states. The app GUI state combines the macroscopic perspective and the microscopic perspective, and attaches the critical semantic information from GUI images. This enables PIRLTest to be platform-independent and makes the testing approach generally applicable on different platforms. PIRLTest explores apps with the guidance of a curiosity-driven strategy, which uses a Q-network to estimate the values of specific state-action pairs to encourage more exploration in uncovered pages without platform dependency. The exploration will be assigned with rewards for all actions, which are designed considering both the app GUI states and the concrete widgets, to help the framework explore more uncovered pages.

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper aims to address several key issues in the automated graphical user interface (GUI) testing of mobile apps and web apps: 1. **Low Code Coverage**: Existing automated testing methods have limitations in achieving high code coverage. 2. **Difficulty in Building High-Quality Models**: Constructing high-quality test models is challenging. 3. **Strong Platform Dependency**: Existing reinforcement learning-based testing methods heavily rely on specific execution platforms (such as Android or Web), resulting in poor adaptability and generality across different platforms. ### Solution To address the above issues, the authors propose a new method called PIRLTest, which achieves platform-independent GUI testing through image embedding and reinforcement learning techniques. Specifically, the main contributions of PIRLTest include: 1. **Platform Independence**: PIRLTest can be adapted to different platforms (such as mobile apps and web apps) at zero cost, without the need for additional modifications or adjustments. 2. **Efficient GUI State Abstraction**: PIRLTest utilizes computer vision techniques to extract GUI widgets and represents the application's GUI state by combining micro perspectives (widget features) and macro perspectives (layout features). 3. **Novel Reward Function Design**: PIRLTest designs a reward function that comprehensively considers both overall GUI exploration rate and specific GUI widget exploration rate to better guide the algorithm in exploring uncovered pages. 4. **Experimental Validation**: Experimental results show that PIRLTest achieves higher code coverage on different platforms, covering 6.3% to 41.4% more code (mobile apps) and 1.5% to 51.1% more code (web apps) than baseline methods. Additionally, PIRLTest can detect 128 unique bugs, 100 of which are undetectable by baseline methods. ### Summary By proposing PIRLTest, the paper addresses the shortcomings of existing automated GUI testing methods in terms of code coverage, model construction, and platform adaptability, providing a new solution to improve software quality.

Effective, Platform-Independent GUI Testing via Image Embedding and Reinforcement Learning

UniRLTest: Universal Platform-Independent Testing with Reinforcement Learning Via Image Understanding

Practical Non-Intrusive GUI Exploration Testing with Visual-based Robotic Arms

A reinforcement learning-based approach to testing GUI of moblie applications

Layout and Image Recognition Driving Cross-Platform Automated Mobile Testing

Deeply Reinforcing Android GUI Testing with Deep Reinforcement Learning.

A Novel Automated GUI Testing Echnology Based on Image Recognition

NiCro: Purely Vision-based, Non-intrusive Cross-Device and Cross-Platform GUI Testing

Visual Identification of Mobile App GUI Elements for Automated Robotic Testing.

Guardian: A Runtime Framework for LLM-based UI Exploration

Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation

Reinforcement Learning Based Curiosity-Driven Testing of Android Applications

Practical, Automated Scenario-based Mobile App Testing

A Unified Framework for Mini-game Testing: Experience on WeChat

Vision-Based Mobile App GUI Testing: A Survey

Practical GUI Testing of Android Applications Via Model Abstraction and Refinement

Appaction: Automatic GUI Interaction for Mobile Apps Via Holistic Widget Perception

Learning-Replay Based Automated Robotic Testing for Mobile App

Automated Mobile App Test Script Intent Generation via Image and Code Understanding

Visual Judgment Approach of Isomorphic GUI for Automated Mobile App Testing

Deep Learning-Based Mobile Application Isomorphic GUI Identification for Automated Robotic Testing