Abstract:Voice-controlled intelligent personal assistants, such as Cortana, Google Now, Siri and Alexa, are increasingly becoming a part of users' daily lives, especially on mobile devices. They introduce a significant change in information access, not only by introducing voice control and touch gestures but also by enabling dialogues where the context is preserved. This raises the need for evaluation of their effectiveness in assisting users with their tasks. However, in order to understand which type of user interactions reflect different degrees of user satisfaction we need explicit judgements. In this paper, we describe a user study that was designed to measure user satisfaction over a range of typical scenarios of use: controlling a device, web search, and structured search dialogue. Using this data, we study how user satisfaction varied with different usage scenarios and what signals can be used for modeling satisfaction in the different scenarios. We find that the notion of satisfaction varies across different scenarios, and show that, in some scenarios (e.g. making a phone call), task completion is very important while for others (e.g. planning a night out), the amount of effort spent is key. We also study how the nature and complexity of the task at hand affects user satisfaction, and find that preserving the conversation context is essential and that overall task-level satisfaction cannot be reduced to query-level satisfaction alone. Finally, we shed light on the relative effectiveness and usefulness of voice-controlled intelligent agents, explaining their increasing popularity and uptake relative to the traditional query-response interaction.

Evaluating Personal Assistants on Mobile devices

Uncovering Post-Adoption Usage of AI-based Voice Assistants: a Technology Affordance Lens Using a Mixed-Methods Approach

Understanding User Satisfaction with Intelligent Assistants

Context-Aware Target Apps Selection and Recommendation for Enhancing Personal Mobile Assistants

Voice Activated Personal Assistant: Acceptability of Use in the Public Space

SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications

Mobile assistant: enhancing desktop interaction using mobile phone

Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)

The Effects of Modality, Device, and Task Differences on Perceived Human Likeness of Voice-Activated Virtual Assistants

Automatically Generating and Improving Voice Command Interface from Operation Sequences on Smartphones

Pardon? An Overview of the Current State and Requirements of Voice User Interfaces for Blind and Visually Impaired Users

Building Proactive Voice Assistants: When and How (not) to Interact

Intelligent Personal Assistant in Business-Context: Key-feature Evaluation for User Acceptance

Challenges in Supporting Exploratory Search through Voice Assistants

A Survey Investigating Usage of Virtual Personal Assistants

Gesture-Based Interactions: Integrating Accelerometer and Gyroscope Sensors in the Use of Mobile Apps

Perceived User Reachability in Mobile UIs Using Data Analytics and Machine Learning

Interaction Techniques for Exploratory Data Visualization on Mobile Devices

Intelligent Assistant Language Understanding On Device

A Systematic Review of Ethical Concerns with Voice Assistants

Comparing the User Preferences Towards Emotional Voice Interaction Applied on Different Devices: An Empirical Study