Abstract:BACKGROUND COVID 2019 restrictions severely curtailed empirical endeavors that involved in-person human interaction. The pandemic also stimulated our team to embrace technology in a two-fold manner. First, we created novel technology to help us to overcome pandemic restrictions in teaching empathic communication in the traditional classroom. A web-based training portal was created for users to learn empathy in an accessible, compelling, self-directed, and interactive online environment. Second, we harnessed technology to engage in remote usability testing and data collection with prospective users of our training portal. In developing our protocol, we discovered gaps in the literature on moderator and silent observer roles and experiences in conducting remote usability testing. OBJECTIVE The aim of this paper is to share our remote moderator and silent observer experiences and their use of certain tools to capture user feedback and experiences with the app. METHODS The larger-scale project employed a quantitative and think-aloud qualitative problem-discovery usability study design. Three trained research assistants collected and utilized user feedback from eight users who were asked to complete tasks in three sessions. Each research assistant had assigned roles and were asked to qualitatively describe their roles, experiences and reactions to the usability testing protocol, and suggestions for improved techniques and strategies for conducting remote usability testing. RESULTS Major results are described in relation to the research assistant experiences with the study protocol followed by recommendations for the design of future remote testing activities as well as evidence-informed training materials for usability project personnel. CONCLUSIONS Our findings highlighted that as we move towards greater remote usability testing, we also need more comprehensive understanding of human-computer interaction and its impact on usability testing outcomes. Our team realized that accessible comprehensive web-conferencing platform to conduct remote sessions is not sufficient. Lead moderator and silent observers offered their insights and recommendations for the ongoing creation and testing of training materials for their respective roles with a focus on: online interpersonal communication skills, conducting user testing protocols, troubleshooting technology and test user issues, proficiency in web-conferencing plus behavior analysis and feedback technologies, and time management. CLINICALTRIAL None.

Sentence Completion as a User Experience Research Method: Recommendations From an Experimental Study

The User in Experimental Computer Systems Research

Using Placeholder Text in Narrative Open-Ended Questions in Web Surveys

Inspiration through Observation: Demonstrating the Influence of Automatically Generated Text on Creative Writing

Consequences and Factors of Stylistic Differences in Human-Robot Dialogue

Editorial Introduction to the Special Issue: Using Story Completion Methods in Qualitative Research

Understanding the Effects of Smart-Speaker-Based Surveys on Panelist Experience in Immersive Consumer Testing

How to Probe Sentence Embeddings in Low-Resource Languages: On Structural Design Choices for Probing Task Evaluation

Suggestion Lists vs. Continuous Generation: Interaction Design for Writing with Generative Models on Mobile Devices Affect Text Length, Wording and Perceived Authorship

Sentence Completion in English Reading Comprehension Tests

Conversation-based hybrid UI for the repertory grid technique: A lab experiment into automation of qualitative surveys

Understanding the Impact that Response Failure has on How Users Perceive Anthropomorphic Conversational Service Agents: Insights from an Online Experiment

"Hit the Robot on the Head With This Mallet" - Making a Case for Including More Open Questions in HRI Research

Towards Low-burden Responses to Open Questions in VR

Use or Consequences: Probing the Cognitive Difference Between Two Measures of Divergent Thinking

A sentence completion procedure as an alternative to the Autobiographical Memory Test for assessing overgeneral memory in non-clinical populations

Moderator and observer experiences and decision-making: Remote usability testing of a web-based empathy training portal (Preprint)

How Social Instant Messaging Questions Affect Replies: a Randomised Controlled Experiment.

Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways

Contextual Response Interpretation for Automated Structured Interviews: A Case Study in Market Research

Positioning the reader: the effect of narrative point-of-view and familiarity of experience on situation model construction