Abstract:Purpose: Usability testing can be used to evaluate human-computer interaction (HCI) and communication in shared decision making (SDM) for patient-provider behavioral change and behavioral contracting. Traditional evaluations of usability using scripted or mock patient scenarios with think-aloud protocol analysis provide a way to identify HCI issues. In this paper we describe the application of these methods in the evaluation of the Avoiding Diabetes Thru Action Plan Targeting (ADAPT) tool, and test the usability of the tool to support the ADAPT framework for integrated care counseling of pre-diabetes. The think-aloud protocol analysis typically does not provide an assessment of how patient-provider interactions are effected in "live" clinical workflow or whether a tool is successful. Therefore, "Near-live" clinical simulations involving applied simulation methods were used to compliment the think-aloud results. This complementary usability technique was used to test the end-user HCI and tool performance by more closely mimicking the clinical workflow and capturing interaction sequences along with assessing the functionality of computer module prototypes on clinician workflow. We expected this method to further complement and provide different usability findings as compared to think-aloud analysis. Together, this mixed method evaluation provided comprehensive and realistic feedback for iterative refinement of the ADAPT system prior to implementation. Methods: The study employed two phases of testing of a new interactive ADAPT tool that embedded an evidence-based shared goal setting component into primary care workflow for dealing with pre-diabetes counseling within a commercial physician office electronic health record (EHR). Phase I applied usability testing that involved "think-aloud" protocol analysis of eight primary care providers interacting with several scripted clinical scenarios. Phase II used "near-live" clinical simulations of five providers interacting with standardized trained patient actors enacting the clinical scenario of counseling for pre-diabetes, each of whom had a pedometer that recorded the number of steps taken over a week. In both phases, all sessions were audio-taped and motion screen-capture software was activated for onscreen recordings. Transcripts were coded using iterative qualitative content analysis methods. Results: In Phase I, the impact of the components and layout of ADAPT on user's Navigation, Understandability, and Workflow were associated with the largest volume of negative comments (i.e. approximately 80% of end-user commentary), while Usability and Content of ADAPT were representative of more positive than negative user commentary. The heuristic category of Usability had a positive-to-negative comment ratio of 2.1, reflecting positive perception of the usability of the tool, its functionality, and overall co-productive utilization of ADAPT. However, there were mixed perceptions about content (i.e., how the information was displayed, organized and described in the tool). In Phase II, the duration of patient encounters was approximately 10 min with all of the Patient Instructions (prescriptions) and behavioral contracting being activated at the end of each visit. Upon activation, providers accepted the pathway prescribed by the tool 100% of the time and completed all the fields in the tool in the simulation cases. Only 14% of encounter time was spent using the functionality of the ADAPT tool in terms of keystrokes and entering relevant data. The rest of the time was spent on communication and dialog to populate the patient instructions. In all cases, the interaction sequence of reviewing and discussing exercise and diet of the patient was linked to the functionality of the ADAPT tool in terms of monitoring, response-efficacy, self-efficacy, and negotiation in the patient-provider dialog. There was a change from one-way dialog to two-way dialog and negotiation that ended in a behavioral contract. This change demonstrated the tool's sequence, which supported recording current exercise and diet followed by a diet and exercise goal setting procedure to reduce the risk of diabetes onset. Conclusions: This study demonstrated that "think-aloud" protocol analysis with "near-live" clinical simulations provided a successful usability evaluation of a new primary care pre-diabetes shared goal setting tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to show the influence of the ADAPT framework on the usability, workflow integration, and communication between the patient and provider. The think-aloud tests with the provider showed the tool can be used according to the ADAPT framework (exercise-to-diet behavior change and tool utilization), while the clinical simulations revealed the ADAPT framework to realistically support patient-provider communication to obtain behavioral change contract. SDM interactions and mechanisms affecting protocol-based care can be more completely captured by combining "near-live" clinical simulations with traditional "think-aloud analysis" which augments clinician utilization. More analysis is required to verify if the rich communication actions found in Phase II compliment clinical workflows.

Rationale and Methods of Evaluation for ACHO, A New Virtual Assistant to Improve Therapeutic Adherence in Rural Elderly Populations: A User-Driven Living Lab

Treatment Adherence in Chronic Conditions during Ageing: Uses, Functionalities, and Cultural Adaptation of the Assistant on Care and Health Offline (ACHO) in Rural Areas

Usability of an Intelligent Virtual Assistant for Promoting Behavior Change and Self-Care in Older People with Type 2 Diabetes

An Audio Personal Health Library of Clinic Visit Recordings for Patients and Their Caregivers (HealthPAL): User-Centered Design Approach

A Patient-Centered Mobile Phone App (iHeartU) With a Virtual Human Assistant for Self-Management of Heart Failure: Protocol for a Usability Assessment Study

Assessment of the Effectiveness, Socio-Economic Impact and Implementation of a Digital Solution for Patients with Advanced Chronic Diseases: The ADLIFE Study Protocol

An User-Centered Evaluation of Two Socially Assistive Robots Integrated in a Retirement Home

A game changer Joint Design Journey of PREMS and PROMS in the HOSMARTAI virtual coach study

Usability of a mobile application for the clinical follow-up of patients with chronic obstructive pulmonary disease and home oxygen therapy

Usability and Utility of a Mobile App to Deliver Health-Related Content to an Older Adult Population: Pilot Noncontrolled Quasi-Experimental Study

A Personalized Voice-Based Diet Assistant for Caregivers of Alzheimer Disease and Related Dementias: System Development and Validation

A Technological Tool Aimed at Self-Care in Patients With Multimorbidity: Cross-Sectional Usability Study

Designing a Social Robot Companion to Support Homecare: Usability Results

Usability of a mobile application for health professionals in home care services: a user-centered approach

Developing and Testing Remote Implementation for the Changing Talk Online (CHATO) Communication Intervention for Nursing Home Staff: A Pilot Pragmatic Randomized Controlled Trial

Co-creation and evaluation of an app to support reminiscence therapy interventions for older people with dementia

Co-designing Strategies to Provide Telecare Through an Intelligent Assistant for Caregivers of Elderly Individuals

Design, implementation and usability analysis of patient empowerment in ADLIFE project via patient reported outcome measures and shared decision making

A VOICE-INTERACTION-MEDIATED SMART HOME TO SUPPORT AGING IN PLACE: A NEW PARADIGM TO ENGAGE WITH HEALTH-RELATED DATA

Usability testing of Avoiding Diabetes Thru Action Plan Targeting (ADAPT) decision support for integrating care-based counseling of pre-diabetes in an electronic health record

Voice Assistants for Health Self-Management: Designing for and with Older Adults