Abstract:Purpose: Usability testing can be used to evaluate human-computer interaction (HCI) and communication in shared decision making (SDM) for patient-provider behavioral change and behavioral contracting. Traditional evaluations of usability using scripted or mock patient scenarios with think-aloud protocol analysis provide a way to identify HCI issues. In this paper we describe the application of these methods in the evaluation of the Avoiding Diabetes Thru Action Plan Targeting (ADAPT) tool, and test the usability of the tool to support the ADAPT framework for integrated care counseling of pre-diabetes. The think-aloud protocol analysis typically does not provide an assessment of how patient-provider interactions are effected in "live" clinical workflow or whether a tool is successful. Therefore, "Near-live" clinical simulations involving applied simulation methods were used to compliment the think-aloud results. This complementary usability technique was used to test the end-user HCI and tool performance by more closely mimicking the clinical workflow and capturing interaction sequences along with assessing the functionality of computer module prototypes on clinician workflow. We expected this method to further complement and provide different usability findings as compared to think-aloud analysis. Together, this mixed method evaluation provided comprehensive and realistic feedback for iterative refinement of the ADAPT system prior to implementation. Methods: The study employed two phases of testing of a new interactive ADAPT tool that embedded an evidence-based shared goal setting component into primary care workflow for dealing with pre-diabetes counseling within a commercial physician office electronic health record (EHR). Phase I applied usability testing that involved "think-aloud" protocol analysis of eight primary care providers interacting with several scripted clinical scenarios. Phase II used "near-live" clinical simulations of five providers interacting with standardized trained patient actors enacting the clinical scenario of counseling for pre-diabetes, each of whom had a pedometer that recorded the number of steps taken over a week. In both phases, all sessions were audio-taped and motion screen-capture software was activated for onscreen recordings. Transcripts were coded using iterative qualitative content analysis methods. Results: In Phase I, the impact of the components and layout of ADAPT on user's Navigation, Understandability, and Workflow were associated with the largest volume of negative comments (i.e. approximately 80% of end-user commentary), while Usability and Content of ADAPT were representative of more positive than negative user commentary. The heuristic category of Usability had a positive-to-negative comment ratio of 2.1, reflecting positive perception of the usability of the tool, its functionality, and overall co-productive utilization of ADAPT. However, there were mixed perceptions about content (i.e., how the information was displayed, organized and described in the tool). In Phase II, the duration of patient encounters was approximately 10 min with all of the Patient Instructions (prescriptions) and behavioral contracting being activated at the end of each visit. Upon activation, providers accepted the pathway prescribed by the tool 100% of the time and completed all the fields in the tool in the simulation cases. Only 14% of encounter time was spent using the functionality of the ADAPT tool in terms of keystrokes and entering relevant data. The rest of the time was spent on communication and dialog to populate the patient instructions. In all cases, the interaction sequence of reviewing and discussing exercise and diet of the patient was linked to the functionality of the ADAPT tool in terms of monitoring, response-efficacy, self-efficacy, and negotiation in the patient-provider dialog. There was a change from one-way dialog to two-way dialog and negotiation that ended in a behavioral contract. This change demonstrated the tool's sequence, which supported recording current exercise and diet followed by a diet and exercise goal setting procedure to reduce the risk of diabetes onset. Conclusions: This study demonstrated that "think-aloud" protocol analysis with "near-live" clinical simulations provided a successful usability evaluation of a new primary care pre-diabetes shared goal setting tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to show the influence of the ADAPT framework on the usability, workflow integration, and communication between the patient and provider. The think-aloud tests with the provider showed the tool can be used according to the ADAPT framework (exercise-to-diet behavior change and tool utilization), while the clinical simulations revealed the ADAPT framework to realistically support patient-provider communication to obtain behavioral change contract. SDM interactions and mechanisms affecting protocol-based care can be more completely captured by combining "near-live" clinical simulations with traditional "think-aloud analysis" which augments clinician utilization. More analysis is required to verify if the rich communication actions found in Phase II compliment clinical workflows.

Realizing the Full Potential of Clinical Decision Support: Translating Usability Testing into Routine Practice in healthcare operations

Mobilizing Clinical Decision Support to Facilitate Knowledge Translation: a Case Study in China.

Usability Testing of Situation Awareness Clinical Decision Support in the ICU

Usability testing of Avoiding Diabetes Thru Action Plan Targeting (ADAPT) decision support for integrating care-based counseling of pre-diabetes in an electronic health record

Applying Clinical Decision Support Design Best Practices With the Practical Robust Implementation and Sustainability Model Versus Reliance on Commercially Available Clinical Decision Support Tools: Randomized Controlled Trial

Applying cognitive walkthrough methodology to improve the usability of an equity-focused implementation strategy

Success Factors for Usability Testing of Digital Health Solutions

What characteristics of clinical decision support system implementations lead to adoption for regular use? A scoping review

Evaluation of User Interface and Workflow Design of a Bedside Nursing Clinical Decision Support System

Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions

Developing a User-Centered Digital Clinical Decision Support App for Evidence-Based Medication Recommendations for Type 2 Diabetes Mellitus: Prototype User Testing and Validation Study

Agile Acceptance Test-Driven Development of Clinical Decision Support Advisories: Feasibility of Using Open Source Software

Developing and Testing the Usability of a Novel Child Abuse Clinical Decision Support System: Mixed Methods Study

A qualitative framework-based evaluation of radiology clinical decision support initiatives: eliciting key factors to physician adoption in implementation

Conducting a Representative National Randomized Control Trial of Tailored Clinical Decision Support for Nurses Remotely: Methods and Implications.

Integrating the Practical Robust Implementation and Sustainability Model With Best Practices in Clinical Decision Support Design: Implementation Science Approach

Formative Usability Evaluation of a Novel Tool for Medication Reconciliation

Healthcare systems collaborating to implement a shared decision-making tool in the electronic health record and build evidence on its adoption and use

Designing and testing clinical simulations of an early warning system for implementation in acute care settings

Integrating Clinical Decision Support Into Electronic Health Record Systems Using a Novel Platform (EvidencePoint): Developmental Study

Usability of an Automated System for Real-Time Monitoring of Shared Decision-Making for Surgery: Mixed Methods Evaluation