Analyzing LLMs' Capabilities to Establish Implicit User Sentiment of Software Desirability

Sherri Weitl-Harms,John D. Hastings,Jonah Lum
2024-08-03
Abstract:This study explores the use of several LLMs for providing quantitative zero-shot sentiment analysis of implicit software desirability expressed by users. The study provides scaled numerical sentiment analysis unlike other methods that simply classify sentiment as positive, neutral, or negative. Numerical analysis provides deeper insights into the magnitude of sentiment, to drive better decisions regarding product desirability. Data is collected through the use of the Microsoft Product Desirability Toolkit (PDT), a well-known qualitative user experience analysis tool. For initial exploration, the PDT metric was given to users of ZORQ, a gamification system used in undergraduate computer science education. The PDT data collected was fed through several LLMs (Claude Sonnet 3 and 3.5, GPT4, and GPT4o) and through a leading transfer learning technique, Twitter-Roberta-Base-Sentiment (TRBS), and through Vader, a leading sentiment analysis tool, for quantitative sentiment analysis. Each system was asked to evaluate the data in two ways, first by looking at the sentiment expressed in the PDT word/explanation pairs; and by looking at the sentiment expressed by the users in their grouped selection of five words and explanations, as a whole. Each LLM was also asked to provide its confidence (low, medium, high) in its sentiment score, along with an explanation of why it selected the sentiment value. All LLMs tested were able to statistically detect user sentiment from the users' grouped data, whereas TRBS and Vader were not. The confidence and explanation of confidence provided by the LLMs assisted in understanding the user sentiment. This study adds to a deeper understanding of evaluating user experiences, toward the goal of creating a universal tool that quantifies implicit sentiment expressed.
Computation and Language,Artificial Intelligence,Human-Computer Interaction,Machine Learning,Software Engineering
What problem does this paper attempt to address?
The problem this paper attempts to address is how to utilize large language models (LLMs) to perform quantitative analysis of implicit emotions expressed by users in software usability surveys. Specifically, the study aims to evaluate the implicit emotions in the data provided by users through the Microsoft Product Desirability Toolkit (PDT) by using several different LLMs (including Claude Sonnet 3 and 3.5, GPT4, and GPT4o) as well as existing sentiment analysis tools (such as Twitter-Roberta-Base-Sentiment (TRBS) and Vader). This data includes five adjectives chosen by users and their explanations to describe their experience with a product. The objectives of the study are: 1. To compare the performance of different LLMs and known sentiment analysis tools in providing sentiment scores for PDT data. 2. To determine whether there is a specific algorithm or combination of algorithms that outperforms others in quantifying implicit user emotions, which could serve as a basis for developing tools for implicit user emotion analysis. Through this approach, the study hopes to bridge the gap between qualitative emotional data and quantitative analysis, extracting meaningful quantitative insights from users' qualitative feedback to better understand users' implicit preferences for products, thereby improving product design and marketing effectiveness.