Abstract:Background: While there is growing demand for information about comparative effectiveness (CE), there is substantial debate about whether and when observational studies have sufficient quality to support decision making. Objective: To develop and test an item checklist that can be used to qualify those observational CE studies sufficiently rigorous in design and execution to contribute meaningfully to the evidence base for decision support. Methods: An 11-item checklist about data and methods (the GRACE checklist) was developed through literature review and consultation with experts from professional societies, payer groups, the private sector, and academia. Since no single gold standard exists for validation, checklist item responses were compared with 3 different types of external quality ratings (N=88 articles). The articles compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. We validated checklist item responses 3 ways against external quality ratings, using published articles of observational CE or safety studies: (a) Systematic Review-quality assessment from a published systematic review; (b) Single Expert Review-quality assessment made according to the solicited "expert opinion" of a senior researcher; and (c) Concordant Expert Review-quality assessments from 2 experts for which there was concordance. Volunteers (N=113) from 5 continents completed 280 article assessments using the checklist. Positive and negative predictive values (PPV, NPV, respectively) of individual items were estimated to compare testers' assessments with those of experts. Results: Taken as a whole, the scale had better NPV than PPV, for both data and methods. The most consistent predictor of quality relates to the validity of the primary outcomes measurement for the study purpose. Other consistent markers of quality relate to using concurrent comparators, minimizing the effects of bias by prudent choice of covariates, and using sensitivity analysis to test robustness of results. Concordance of expert opinion on the quality of the rated articles was 52%; most checklist items performed better. Conclusions: The 11-item GRACE checklist provides guidance to help determine which observational studies of CE have used strong scientific methods and good data that are fit for purpose and merit consideration for decision making. The checklist contains a parsimonious set of elements that can be objectively assessed in published studies, and user testing shows that it can be successfully applied to studies of drugs, medical devices, and clinical and surgical interventions. Although no scoring is provided, study reports that rate relatively well across checklist items merit in-depth examination to understand applicability, effect size, and likelihood of residual bias. The current testing and validation efforts did not achieve clear discrimination between studies fit for purpose and those not, but we have identified a critical, though remediable, limitation in our approach. Not specifying a specific granular decision for evaluation, or not identifying a single study objective in reports that included more than one, left reviewers with too broad an assessment challenge. We believe that future efforts will be more successful if reviewers are asked to focus on a specific objective or question. Despite the challenges encountered in this testing, an agreed upon set of assessment elements, checklists, or score cards is critical for the maturation of this field. Substantial resources will be expended on studies of real-world effectiveness, and if the rigor of these observational assessments cannot be assessed, then the impact of the studies will be suboptimal. Similarly, agreement on key elements of quality will ensure that budgets are appropriately directed toward those elements. Given the importance of this task and the lessons learned from these extensive efforts at validation and user testing, we are optimistic about the potential for improved assessments that can be used for diverse situations by people with a wide range of experience and training. Future testing would benefit by directing reviewers to address a single, granular research question, which would avoid problems that arose by using the checklist to evaluate multiple objectives, by using other types of validation test sets, and by employing further multivariate analysis to see if any combination or sequence of item responses has particularly high predictive validity.

Comparing regulatory guidance on risk minimization/mitigation and the Reporting recommendation Intended for pharmaceutical Risk Minimization Evaluation Studies checklist

The Reporting Recommendations Intended for Pharmaceutical Risk Minimization Evaluation Studies: Standards for Reporting of Implementation Studies Extension (RIMES-SE)

Are drug safety measures harmonized? A real-world evidence from three regulatory authorities

Limitations Reported in Evaluating Effectiveness of Risk Minimization Measures in the EU during 2018-2021: A Qualitative Analysis of Industry-Sponsored Post-Authorization Safety Studies

[Risk minimalisation measures for medications; are they incorporated in Dutch clinical guidelines?]

Comparison of published core outcome sets with outcomes recommended in regulatory guidance from the US Food and Drug Administration and European Medicines Agency: cross sectional analysis

Comprehensive Assessment of Risk-Based Quality Management Adoption in Clinical Trials

Advancing the Field of Pharmaceutical Risk Minimization Through Application of Implementation Science Best Practices

Guideline for reporting systematic reviews of outcome measurement instruments (OMIs): PRISMA-COSMIN for OMIs 2024

An analysis of international strategies for monitoring and preventability described in medicinal product information: a study protocol

The REporting of A Disproportionality Analysis for DrUg Safety Signal Detection Using Individual Case Safety Reports in PharmacoVigilance (READUS-PV): Explanation and Elaboration

Research report on guidance for the multi-regional clinical trials

Contemporary Practice and Considerations for Real-World Data Source Identification and Feasibility Assessment

Reporting Quality in Health Economic Evaluation Studies of Immune Checkpoint Inhibitors: A Systematic Review

EMA and FDA psychiatric drug trial guidelines: assessment of guideline development and trial design recommendations

The Reporting of a Disproportionality Analysis for Drug Safety Signal Detection Using Individual Case Safety Reports in PharmacoVigilance (READUS-PV): Development and Statement

Guidance on risk-benefit assessment of foods

Reporting Guidelines for Clinical Pharmacokinetic Studies: The ClinPK Statement

Assessment of pharmacovigilance guidelines in the Southern African Development Community: A document review

The GRACE checklist for rating the quality of observational studies of comparative effectiveness: a tale of hope and caution

The State of Patient-Reported Outcome Measures in Rheumatology