Abstract:Mobile applications (apps) have become an essential part of our daily lives, making ensuring their quality an important activity. GUI testing, a quality assurance method, has frequently been used for mobile apps. When conducting GUI testing, it is important to generate effective text inputs for the text-input components. Some GUIs require these text inputs to move from one page to the next, which remains a challenge to achieving complete UI exploration. Recently, Large Language Models (LLMs) have shown excellent text-generation capabilities. Among the LLMs, OpenAI's GPT series has been widely discussed and used. However, it may not be possible to use these LLMs for GUI testing actual mobile apps, due to the security and privacy issues related to the production data. Therefore, it is necessary to explore the potential of different LLMs to guide text-input generation in mobile GUI testing. This paper reports on a large-scale empirical study that extensively investigates the effectiveness of nine state-of-the-art LLMs in Android text-input generation for UI pages. We collected 114 UI pages from 62 open-source Android apps and extracted contextual information from the UI pages to construct prompts for LLMs to generate text inputs. The experimental results show that some LLMs can generate relatively more effective and higher-quality text inputs, achieving a 50.58% to 66.67% page-pass-through rate, and even detecting some real bugs in open-source apps. Compared with the GPT-3.5 and GPT-4 LLMs, other LLMs reduce the page-pass-through rates by 17.97% to 84.79% and 21.93% to 85.53%, respectively. We also found that using more complete UI contextual information can increase the page-pass-through rates of LLMs for generating text inputs. In addition, we also describe six insights gained regarding the use of LLMs for Android testing: These insights will benefit the Android testing community.

Does GenAI Make Usability Testing Obsolete?

SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications

Predicting the usability of mobile applications using AI tools: the rise of large user interface models, opportunities, and challenges

Usability Evaluation of Augmented Reality: A Neuro-Information-Systems Study.

Large Language Models for Mobile GUI Text Input Generation: An Empirical Study

The fine line between automation and augmentation in website usability evaluation

Design and Implementation of a Toolkit for Usability Testing of Mobile Apps

Establishing Heuristics for Improving the Usability of GUI Machine Learning Tools for Novice Users

Usability Heuristic Evaluation for the Hearing Impaired Language Training Mobile App

AXNav: Replaying Accessibility Tests from Natural Language

Enhancing UX Research Activities Using GenAI -- Potential Applications and Challenges

Evaluating the Usability of LLMs in Threat Intelligence Enrichment

On AI-Inspired UI-Design

Mobile language learning applications for Arabic speaking migrants – a usability perspective

User Guided Automation for Testing Mobile Apps

A Genetic Algorithm-Based Support Vector Machine Approach for Intelligent Usability Assessment of m-Learning Applications

In Situ AI Prototyping: Infusing Multimodal Prompts into Mobile Settings with MobileMaker

Automated Test Transfer Across Android Apps Using Large Language Models

EvAlignUX: Advancing UX Research through LLM-Supported Exploration of Evaluation Metrics

Human-AI Collaboration for UX Evaluation: Effects of Explanation and Synchronization

Addressing UX Practitioners' Challenges in Designing ML Applications: an Interactive Machine Learning Approach