Abstract:Purpose: Usability testing can be used to evaluate human-computer interaction (HCI) and communication in shared decision making (SDM) for patient-provider behavioral change and behavioral contracting. Traditional evaluations of usability using scripted or mock patient scenarios with think-aloud protocol analysis provide a way to identify HCI issues. In this paper we describe the application of these methods in the evaluation of the Avoiding Diabetes Thru Action Plan Targeting (ADAPT) tool, and test the usability of the tool to support the ADAPT framework for integrated care counseling of pre-diabetes. The think-aloud protocol analysis typically does not provide an assessment of how patient-provider interactions are effected in "live" clinical workflow or whether a tool is successful. Therefore, "Near-live" clinical simulations involving applied simulation methods were used to compliment the think-aloud results. This complementary usability technique was used to test the end-user HCI and tool performance by more closely mimicking the clinical workflow and capturing interaction sequences along with assessing the functionality of computer module prototypes on clinician workflow. We expected this method to further complement and provide different usability findings as compared to think-aloud analysis. Together, this mixed method evaluation provided comprehensive and realistic feedback for iterative refinement of the ADAPT system prior to implementation. Methods: The study employed two phases of testing of a new interactive ADAPT tool that embedded an evidence-based shared goal setting component into primary care workflow for dealing with pre-diabetes counseling within a commercial physician office electronic health record (EHR). Phase I applied usability testing that involved "think-aloud" protocol analysis of eight primary care providers interacting with several scripted clinical scenarios. Phase II used "near-live" clinical simulations of five providers interacting with standardized trained patient actors enacting the clinical scenario of counseling for pre-diabetes, each of whom had a pedometer that recorded the number of steps taken over a week. In both phases, all sessions were audio-taped and motion screen-capture software was activated for onscreen recordings. Transcripts were coded using iterative qualitative content analysis methods. Results: In Phase I, the impact of the components and layout of ADAPT on user's Navigation, Understandability, and Workflow were associated with the largest volume of negative comments (i.e. approximately 80% of end-user commentary), while Usability and Content of ADAPT were representative of more positive than negative user commentary. The heuristic category of Usability had a positive-to-negative comment ratio of 2.1, reflecting positive perception of the usability of the tool, its functionality, and overall co-productive utilization of ADAPT. However, there were mixed perceptions about content (i.e., how the information was displayed, organized and described in the tool). In Phase II, the duration of patient encounters was approximately 10 min with all of the Patient Instructions (prescriptions) and behavioral contracting being activated at the end of each visit. Upon activation, providers accepted the pathway prescribed by the tool 100% of the time and completed all the fields in the tool in the simulation cases. Only 14% of encounter time was spent using the functionality of the ADAPT tool in terms of keystrokes and entering relevant data. The rest of the time was spent on communication and dialog to populate the patient instructions. In all cases, the interaction sequence of reviewing and discussing exercise and diet of the patient was linked to the functionality of the ADAPT tool in terms of monitoring, response-efficacy, self-efficacy, and negotiation in the patient-provider dialog. There was a change from one-way dialog to two-way dialog and negotiation that ended in a behavioral contract. This change demonstrated the tool's sequence, which supported recording current exercise and diet followed by a diet and exercise goal setting procedure to reduce the risk of diabetes onset. Conclusions: This study demonstrated that "think-aloud" protocol analysis with "near-live" clinical simulations provided a successful usability evaluation of a new primary care pre-diabetes shared goal setting tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to show the influence of the ADAPT framework on the usability, workflow integration, and communication between the patient and provider. The think-aloud tests with the provider showed the tool can be used according to the ADAPT framework (exercise-to-diet behavior change and tool utilization), while the clinical simulations revealed the ADAPT framework to realistically support patient-provider communication to obtain behavioral change contract. SDM interactions and mechanisms affecting protocol-based care can be more completely captured by combining "near-live" clinical simulations with traditional "think-aloud analysis" which augments clinician utilization. More analysis is required to verify if the rich communication actions found in Phase II compliment clinical workflows.

Systematic Improvement of a Bolus Calculator That Is on the Market: A User-Centric Approach

Advancing Diabetes Self-Management: A Novel Smartphone Application Featuring a Scoring Algorithm for Tailored User Engagement

1824-LB: A Qualitative Analysis of the DM-BOOST Program—Enhancing Patient Engagement and Experience with Diabetes Self-Management Education and Support

User-Centered Design of A Novel Risk Prediction Behavior Change Tool Augmented With an Artificial Intelligence Engine (MyDiabetesIQ): A Sociotechnical Systems Approach

Developing a User-Centered Digital Clinical Decision Support App for Evidence-Based Medication Recommendations for Type 2 Diabetes Mellitus: Prototype User Testing and Validation Study

DiaFit: the Development of a Smart App for Patients with Type 2 Diabetes and Obesity

Insights into the Feasibility and Acceptability of a Mobile Insulin Titration Application in Clinical Practice and Its Effects on Diabetes Outcomes

Barriers and Enablers to the Adoption of a Healthier Diet Using an App: Qualitative Interview Study With Patients With Type 2 Diabetes Mellitus

Usability testing of Avoiding Diabetes Thru Action Plan Targeting (ADAPT) decision support for integrating care-based counseling of pre-diabetes in an electronic health record

Empowering patients to address diabetes care gaps: formative usability testing of a novel patient portal intervention

Augmenting clinicians' analytical workflow through task-based integration of data visualizations and algorithmic insights: a user-centered design study

Diabetes patients' experiences with the implementation of insulin therapy and their perceptions of computer-assisted self-management systems for insulin therapy

Digital Health Tools and Behavioral Strategies to Increase Engagement With Diabetes Self-Management Education and Support: Design and Feasibility of DM-BOOST

Heuristic Evaluation of a Top-Rated Diabetes Self-Management App

Usability and Preliminary Efficacy of an Artificial Intelligence-Driven Platform Supporting Dietary Management in Diabetes: Mixed Methods Study

Systematic Evaluation of Canadian Diabetes Smartphone Applications for People With Type 1, Type 2 and Gestational Diabetes

User-Centered Design of a Clinical Tool for Shared Decision-making About Diet in Primary Care

Evaluation of User Interface and Workflow Design of a Bedside Nursing Clinical Decision Support System

Developing the UVA/Padova Type 1 Diabetes Simulator: Modeling, Validation, Refinements, and Utility

App Design Features Important for Diabetes Self-management as Determined by the Self-Determination Theory on Motivation: Content Analysis of Survey Responses From Adults Requiring Insulin Therapy

Translating What Works: A New Approach to Improve Diabetes Management