Abstract:Background: While there is growing demand for information about comparative effectiveness (CE), there is substantial debate about whether and when observational studies have sufficient quality to support decision making. Objective: To develop and test an item checklist that can be used to qualify those observational CE studies sufficiently rigorous in design and execution to contribute meaningfully to the evidence base for decision support. Methods: An 11-item checklist about data and methods (the GRACE checklist) was developed through literature review and consultation with experts from professional societies, payer groups, the private sector, and academia. Since no single gold standard exists for validation, checklist item responses were compared with 3 different types of external quality ratings (N=88 articles). The articles compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. We validated checklist item responses 3 ways against external quality ratings, using published articles of observational CE or safety studies: (a) Systematic Review-quality assessment from a published systematic review; (b) Single Expert Review-quality assessment made according to the solicited "expert opinion" of a senior researcher; and (c) Concordant Expert Review-quality assessments from 2 experts for which there was concordance. Volunteers (N=113) from 5 continents completed 280 article assessments using the checklist. Positive and negative predictive values (PPV, NPV, respectively) of individual items were estimated to compare testers' assessments with those of experts. Results: Taken as a whole, the scale had better NPV than PPV, for both data and methods. The most consistent predictor of quality relates to the validity of the primary outcomes measurement for the study purpose. Other consistent markers of quality relate to using concurrent comparators, minimizing the effects of bias by prudent choice of covariates, and using sensitivity analysis to test robustness of results. Concordance of expert opinion on the quality of the rated articles was 52%; most checklist items performed better. Conclusions: The 11-item GRACE checklist provides guidance to help determine which observational studies of CE have used strong scientific methods and good data that are fit for purpose and merit consideration for decision making. The checklist contains a parsimonious set of elements that can be objectively assessed in published studies, and user testing shows that it can be successfully applied to studies of drugs, medical devices, and clinical and surgical interventions. Although no scoring is provided, study reports that rate relatively well across checklist items merit in-depth examination to understand applicability, effect size, and likelihood of residual bias. The current testing and validation efforts did not achieve clear discrimination between studies fit for purpose and those not, but we have identified a critical, though remediable, limitation in our approach. Not specifying a specific granular decision for evaluation, or not identifying a single study objective in reports that included more than one, left reviewers with too broad an assessment challenge. We believe that future efforts will be more successful if reviewers are asked to focus on a specific objective or question. Despite the challenges encountered in this testing, an agreed upon set of assessment elements, checklists, or score cards is critical for the maturation of this field. Substantial resources will be expended on studies of real-world effectiveness, and if the rigor of these observational assessments cannot be assessed, then the impact of the studies will be suboptimal. Similarly, agreement on key elements of quality will ensure that budgets are appropriately directed toward those elements. Given the importance of this task and the lessons learned from these extensive efforts at validation and user testing, we are optimistic about the potential for improved assessments that can be used for diverse situations by people with a wide range of experience and training. Future testing would benefit by directing reviewers to address a single, granular research question, which would avoid problems that arose by using the checklist to evaluate multiple objectives, by using other types of validation test sets, and by employing further multivariate analysis to see if any combination or sequence of item responses has particularly high predictive validity.

An expert consensus–based checklist for quality appraisal of educational resources on adult basic life support: a Delphi study

Validation of a detailed scoring checklist for use during advanced cardiac life support certification

Development of an Emergency Department Safety Checklist through a global consensus process

Healthcare professionals' knowledge of the systematic ABCDE approach: a cross-sectional study

Development of a simple, practice-based tool to assess quality of paediatric emergency care delivery in resource-limited settings: identifying critical actions via a Delphi study

Validation of a checklist to facilitate serious illness conversations in adult emergency in China: a single-centre pilot study

Evaluating Quality, Usability, Evidence-Based Content, and Gamification Features in Mobile Learning Apps Designed to Teach Children Basic Life Support: Systematic Search in App Stores and Content Analysis

Emergency Medicine and Critical Care Blogs and Podcasts: Establishing an International Consensus on Quality

Construction and psychometric validation in Spanish schoolchildren of a knowledge questionnaire on basic life support and Automated External Defibrillator (ConocES-BLS/AED) in Spain

A consensus-based tool for capability benchmarking of emergency medical services in South Africa

[Cardiac arrest in school: the role of teachers as BLS-D providers]

Emergency Department Thoracotomy: Development of a Reliable, Validated Checklist for Procedural Training

2021 International Consensus on Cardiopulmonary Resuscitation and Emergency Cardiovascular Care Science With Treatment Recommendations

Effectiveness of Adult Chest Compressions during Resuscitation Performed by Children Aged 10–14 Years under Simulated Conditions

Factors Associated With Improved Pediatric Resuscitative Care in General Emergency Departments

Methodologic Quality of Guidelines for Training or Competence Processes for Basic Point-of-Care Echocardiography in Critical Care: A Systematic Review of the Literature

Life supporting first aid training of the public—review and recommendations

Responding to the real problem of sustainable resuscitation skills with real assessment. Mixed‐methods evaluation of an authentic assessment programme

Content validity and inter-rater reliability of procedural skill checklists used in the online OSCE scoring management system

The GRACE checklist for rating the quality of observational studies of comparative effectiveness: a tale of hope and caution