Using ChatGPT-4 for the Identification of Common UX Factors within a Pool of Measurement Items from Established UX Questionnaires

Stefan Graser,Stephan Böhm,Martin Schrepp
2024-11-20
Abstract:Measuring User Experience (UX) with standardized questionnaires is a widely used method. A questionnaire is based on different scales that represent UX factors and items. However, the questionnaires have no common ground concerning naming different factors and the items used to measure them. This study aims to identify general UX factors based on the formulation of the measurement items. Items from a set of 40 established UX questionnaires were analyzed by Generative AI (GenAI) to identify semantically similar items and to cluster similar topics. We used the LLM ChatGPT-4 for this analysis. Results show that ChatGPT-4 can classify items into meaningful topics and thus help to create a deeper understanding of the structure of the UX research field. In addition, we show that ChatGPT-4 can filter items related to a predefined UX concept out of a pool of UX items.
Human-Computer Interaction
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is how to identify and integrate common UX factors from multiple established user experience (UX) questionnaires. Specifically, the paper aims to identify semantically similar items by analyzing the measurement items in these questionnaires and cluster them into common themes or factors. This helps to establish a unified understanding and naming convention among different UX questionnaires, thereby better understanding the structure of the UX research field. ### Main Research Questions: 1. **RQ1: Can generative AI identify useful similar themes based on measurement items?** - Researchers hope to use generative AI (such as ChatGPT - 4) to analyze the measurement items in different UX questionnaires, identify semantically similar items, and cluster them into meaningful themes. 2. **RQ2: What themes can be identified based on semantically similar measurement items in the most commonly used UX questionnaires?** - By analyzing the measurement items in multiple established UX questionnaires, researchers hope to find the common UX factors or themes represented by these items. ### Background and Motivation: - **Diversity of UX Questionnaires**: Although there are many standardized UX questionnaires, they lack consistency in naming and specific measurement items. Different questionnaires may use different names to represent the same UX factor, or use the same name to represent different factors. - **Importance of Semantic Similarity**: In order to better understand the structure of UX, researchers need to analyze the semantic similarity of measurement items to identify common UX factors. - **Application of Generative AI**: The paper uses large - language models (LLM) such as ChatGPT - 4 to conduct semantic text similarity analysis in the hope of discovering a deeper UX structure. ### Methods: - **Data Collection**: Researchers collected 408 measurement items from 40 established UX questionnaires. - **Generative AI Analysis**: Use ChatGPT - 4 to classify and cluster these items, and gradually refine the classification results through a series of prompts. - **Comparison and Verification**: Compare the AI - generated classification results with the existing list of UX factors to verify their rationality and consistency. ### Results: - **Preliminary Classification**: ChatGPT - 4 divided the measurement items into 6 main themes, such as usability, design, user participation, etc. - **Detailed Classification**: After further refinement, 10 more specific themes were obtained, such as ease of use, complexity issues, design appearance, etc. - **Extended Classification**: Continued subdivision yielded 22 sub - themes, covering more specific aspects of UX. - **Improved Classification**: Finally, ChatGPT - 4 proposed 6 main themes and 16 sub - themes, showing a more reasonable classification structure. - **Comparison with Existing Concepts**: The AI - generated classification results were compared with the existing 16 UX quality aspects, and some consistencies and differences were verified. Through these steps, researchers hope to provide a more unified and clear factor framework for the UX research field, helping researchers and practitioners better understand and apply UX measurement tools.