Abstract:Digital interventions often suffer from low usage, which may reflect insufficient attention to user experience. Moreover, the existing evaluation methods have limited applicability in the remote study of user experience of complex interventions that have expansive content and that are used over an extensive period of time. To alleviate these challenges, we describe here a novel qualitative Ecological Momentary Assessment (EMA) method: the CORTO method (Contextual, One-item, Repeated, Timely, Open-ended). We used it to gather digital intervention user experience data from Finnish adults ( n = 184) who lived with interview-confirmed major depressive disorder (MDD) and took part in a randomized controlled trial (RCT) that studied the efficacy of a novel 12-week game-based digital intervention for depression. A second dataset on user experience was gathered with retrospective interviews ( n = 22). We inductively coded the CORTO method and retrospective interview data, which led to four user experience categories: (1) contextual use, (2) interaction-elicited emotional experience, (3) usability, and (4) technical issues. Then, we used the created categorization and Template Analysis to analyze both datasets together, and reported the results qualitatively. Finally, we compared the two datasets with each other. We found that the data generated with the CORTO method offered more insights into usability and technical categories than the interview data that particularly illustrated the contextual use. The emotional valence of the interview data was more positive compared with the CORTO data. Both the CORTO and interview data detected 55 % of the micro-level categories; 20 % of micro-level categories were only detected by the CORTO data and 25 % only by the interview data. We found that the during-intervention user experience measurement with the CORTO method can provide intervention-specific insights, and thereby further the iterative user-centered intervention development. Overall, these findings highlight the impact of evaluation methods on the categories and qualities of insights acquired in intervention research.

CORAE: A Tool for Intuitive and Continuous Retrospective Evaluation of Interactions

"How Did They Come Across?" Lessons Learned from Continuous Affective Ratings

What Predicts Interpersonal Affect? Preliminary Analyses from Retrospective Evaluations

The AffectToolbox: Affect Analysis for Everyone

Measuring digital intervention user experience with a novel ecological momentary assessment (EMA) method, CORTO

COCTEAU: an Empathy-Based Tool for Decision-Making

REACT: Two Datasets for Analyzing Both Human Reactions and Evaluative Feedback to Robots Over Time

The OMG-Empathy Dataset: Evaluating the Impact of Affective Behavior in Storytelling

The uulmMAC Database—A Multimodal Affective Corpus for Affective Computing in Human-Computer Interaction

EMP-EVAL: A Framework for Measuring Empathy in Open Domain Dialogues

An augmented reality interface to control a collaborative robot in rehab: A preliminary usability evaluation

The BIRAFFE2 Experiment. Study in Bio-Reactions and Faces for Emotion-based Personalization for AI Systems

Corona Health -- A Study- and Sensor-based Mobile App Platform Exploring Aspects of the COVID-19 Pandemic

The Affective Dynamics of Everyday Digital Life: Opening Computational Possibility

“I Didn’t Know I Looked Angry”: Characterizing Observed Emotion and Reported Affect at Work

K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations

Comparing Continuous and Retrospective Emotion Ratings in Remote VR Study

BigEAR: Inferring the Ambient and Emotional Correlates from Smartphone-based Acoustic Big Data

A Survey on Identifying Empathic Expression in Remote Collaboration From Empathic Computing

Development and Validation of Pictographic Scales for Rapid Assessment of Affective States in Virtual Reality

These Aren't The Droids You Are Looking for: Promises and Challenges for the Intersection of Affective Science and Robotics/AI