Abstract:Background It is well known that recommendations from electronic medication alerts are seldom accepted or acted on by users. Key factors affecting the effectiveness of medication alerts include system usability and alert design. Thus, human factors principles that apply knowledge of human capabilities and limitations are increasingly used in the design of health technology to improve the usability of systems. Objective This study aims to evaluate a newly developed evidence-based self-assessment tool that allows the valid and reliable evaluation of computerized medication alerting systems. This tool was developed to be used by hospital staff with detailed knowledge of their hospital’s computerized provider order entry system and alerts to identify and address potential system deficiencies. In this initial assessment, we aim to determine whether the items in the tool can measure compliance of medication alerting systems with human factors principles of design, the tool can be consistently used by multiple users to assess the same system, and the items are easy to understand and perceived to be useful for assessing medication alerting systems. Methods The Tool for Evaluating Medication Alerting Systems (TEMAS) was developed based on human factors design principles and consisted of 66 items. In total, 18 staff members recruited across 6 hospitals used the TEMAS to assess their medication alerting systems. Data collected from participant assessments were used to evaluate the validity, reliability, and usability of the TEMAS. Validity was assessed by comparing the results of the TEMAS with those of prior in-house evaluations. Reliability was measured using Krippendorff α to determine agreement among assessors. A 7-item survey was used to determine usability. Results The participants reported mostly negative (n=8) and neutral (n=7) perceptions of alerts in their medication alerting system. However, the validity of the TEMAS could not be directly tested, as participants were unaware of any results from prior in-house evaluations. The reliability of the TEMAS, as measured by Krippendorff α, was low to moderate (range 0.26-0.46); however, participant feedback suggests that individuals’ knowledge of the system varied according to their professional background. In terms of usability, 61% (11/18) of participants reported that the TEMAS items were generally easy to understand; however, participants suggested the revision of 22 items to improve clarity. Conclusions This initial assessment of the TEMAS allowed the identification of its components that required modification to improve usability and usefulness. It also revealed that for the TEMAS to be effective in facilitating a comprehensive assessment of a medication alerting system, it should be completed by a multidisciplinary team of hospital staff from both clinical and technical backgrounds to maximize their knowledge of systems.

Development of a trigger tool to identify harmful incidents, no harm incidents, and near misses in prehospital emergency care

Adaption of a trigger tool to identify harmful incidents, no harm incidents, and near misses in prehospital emergency care of children

Registered nurses' experiences of near misses in ambulance care - A critical incident technique study

Development, Testing, and Findings of a Pediatric-Focused Trigger Tool to Identify Medication-Related Harm in US Children's Hospitals

The emergency department trigger tool: Multicenter trigger query validation

Patient safety and patient assessment in pre-hospital care: a study protocol

Development of a trigger tool for the detection of adverse drug events in Chinese geriatric inpatients using the Delphi method

Measuring the overall development of patient safety in a new hospital using trigger tools

Using the electronic health record to build a culture of practice safety: evaluating the implementation of trigger tools in one general practice.

Towards the development of a screening tool to enhance the detection of elder abuse and neglect by emergency medical technicians (EMTs): a qualitative study

"Near Miss": A Mixed-Methods Analysis of Medical Student Assignments in Patient Safety

Development of Automated Triggers in Ambulatory Settings in Brazil: Protocol for a Machine Learning-Based Design Thinking Study

Type of Track and Trigger system and incidence of in-hospital cardiac arrest: an observational registry-based study

Occurrence of No-Harm Incidents and Adverse Events in Hospitalized Patients with Ischemic Stroke or TIA: A Cohort Study Using Trigger Tool Methodology

Evaluation of an automated surveillance system using trigger alerts to prevent adverse drug events in the intensive care unit and general ward

An electronic trigger to detect telemedicine-related diagnostic errors

Use of ADE Trigger Tool for Monitoring Adverse Drug Events

Behind the scenes: a qualitative study on threats and violence in emergency medical services

Investigation of Medical Accidents for the Development of Alert System to Reduce Alert Fatigue

Monitoring Diagnostic Safety Risks in Emergency Departments: Protocol for a Machine Learning Study

A Tool for Evaluating Medication Alerting Systems: Development and Initial Assessment