Abstract:Background: While there is growing demand for information about comparative effectiveness (CE), there is substantial debate about whether and when observational studies have sufficient quality to support decision making. Objective: To develop and test an item checklist that can be used to qualify those observational CE studies sufficiently rigorous in design and execution to contribute meaningfully to the evidence base for decision support. Methods: An 11-item checklist about data and methods (the GRACE checklist) was developed through literature review and consultation with experts from professional societies, payer groups, the private sector, and academia. Since no single gold standard exists for validation, checklist item responses were compared with 3 different types of external quality ratings (N=88 articles). The articles compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. We validated checklist item responses 3 ways against external quality ratings, using published articles of observational CE or safety studies: (a) Systematic Review-quality assessment from a published systematic review; (b) Single Expert Review-quality assessment made according to the solicited "expert opinion" of a senior researcher; and (c) Concordant Expert Review-quality assessments from 2 experts for which there was concordance. Volunteers (N=113) from 5 continents completed 280 article assessments using the checklist. Positive and negative predictive values (PPV, NPV, respectively) of individual items were estimated to compare testers' assessments with those of experts. Results: Taken as a whole, the scale had better NPV than PPV, for both data and methods. The most consistent predictor of quality relates to the validity of the primary outcomes measurement for the study purpose. Other consistent markers of quality relate to using concurrent comparators, minimizing the effects of bias by prudent choice of covariates, and using sensitivity analysis to test robustness of results. Concordance of expert opinion on the quality of the rated articles was 52%; most checklist items performed better. Conclusions: The 11-item GRACE checklist provides guidance to help determine which observational studies of CE have used strong scientific methods and good data that are fit for purpose and merit consideration for decision making. The checklist contains a parsimonious set of elements that can be objectively assessed in published studies, and user testing shows that it can be successfully applied to studies of drugs, medical devices, and clinical and surgical interventions. Although no scoring is provided, study reports that rate relatively well across checklist items merit in-depth examination to understand applicability, effect size, and likelihood of residual bias. The current testing and validation efforts did not achieve clear discrimination between studies fit for purpose and those not, but we have identified a critical, though remediable, limitation in our approach. Not specifying a specific granular decision for evaluation, or not identifying a single study objective in reports that included more than one, left reviewers with too broad an assessment challenge. We believe that future efforts will be more successful if reviewers are asked to focus on a specific objective or question. Despite the challenges encountered in this testing, an agreed upon set of assessment elements, checklists, or score cards is critical for the maturation of this field. Substantial resources will be expended on studies of real-world effectiveness, and if the rigor of these observational assessments cannot be assessed, then the impact of the studies will be suboptimal. Similarly, agreement on key elements of quality will ensure that budgets are appropriately directed toward those elements. Given the importance of this task and the lessons learned from these extensive efforts at validation and user testing, we are optimistic about the potential for improved assessments that can be used for diverse situations by people with a wide range of experience and training. Future testing would benefit by directing reviewers to address a single, granular research question, which would avoid problems that arose by using the checklist to evaluate multiple objectives, by using other types of validation test sets, and by employing further multivariate analysis to see if any combination or sequence of item responses has particularly high predictive validity.

The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomised Studies in Meta-Analyses

Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments

The ROBINS‐I and the NOS had similar reliability but differed in applicability: A random sampling observational studies of systematic reviews/meta‐analysis

Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis

The GRACE checklist for rating the quality of observational studies of comparative effectiveness: a tale of hope and caution

Evaluating the Effectiveness of Indigenous Health Curricula: Validation and Application of the NOSM CAST Instrument

Reliability testing of the Health of the Nation Outcome Scales 2018

Cross-Cultural Validation of the Portuguese Version of the Quality of Oncology Nursing Care Scale

Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review

Systematic review of applications and properties of the NoMAD instrument for assessing implementation outcomes: Study protocol

The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta-analysis, and clinical practice guideline: a systematic review.

Introduction and Systematic Review of the Good Nursing Care Scale

Ten Points for High-Quality Statistical Reporting and Data Presentation

SANRA—a scale for the quality assessment of narrative review articles

Guideline for reporting systematic reviews of outcome measurement instruments (OMIs): PRISMA-COSMIN for OMIs 2024

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

Comparing and Validating Simple Measures of Patient-Reported Peripheral Neuropathy for Oncology Clinical Trials: NCCTG N0897 (Alliance) A Pooled Analysis of 2440 Patients

Establishing nursing-sensitive quality indicators for pediatric oncology: An international mixed methods Delphi study

Pilot Study of the International Parkinson and Movement Disorder Society‐sponsored Non‐motor Rating Scale (MDS‐NMS)

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project

The neonatal adverse event severity scale: current status, a stakeholders' assessment, and future perspectives