Abstract:Background: While there is growing demand for information about comparative effectiveness (CE), there is substantial debate about whether and when observational studies have sufficient quality to support decision making. Objective: To develop and test an item checklist that can be used to qualify those observational CE studies sufficiently rigorous in design and execution to contribute meaningfully to the evidence base for decision support. Methods: An 11-item checklist about data and methods (the GRACE checklist) was developed through literature review and consultation with experts from professional societies, payer groups, the private sector, and academia. Since no single gold standard exists for validation, checklist item responses were compared with 3 different types of external quality ratings (N=88 articles). The articles compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. We validated checklist item responses 3 ways against external quality ratings, using published articles of observational CE or safety studies: (a) Systematic Review-quality assessment from a published systematic review; (b) Single Expert Review-quality assessment made according to the solicited "expert opinion" of a senior researcher; and (c) Concordant Expert Review-quality assessments from 2 experts for which there was concordance. Volunteers (N=113) from 5 continents completed 280 article assessments using the checklist. Positive and negative predictive values (PPV, NPV, respectively) of individual items were estimated to compare testers' assessments with those of experts. Results: Taken as a whole, the scale had better NPV than PPV, for both data and methods. The most consistent predictor of quality relates to the validity of the primary outcomes measurement for the study purpose. Other consistent markers of quality relate to using concurrent comparators, minimizing the effects of bias by prudent choice of covariates, and using sensitivity analysis to test robustness of results. Concordance of expert opinion on the quality of the rated articles was 52%; most checklist items performed better. Conclusions: The 11-item GRACE checklist provides guidance to help determine which observational studies of CE have used strong scientific methods and good data that are fit for purpose and merit consideration for decision making. The checklist contains a parsimonious set of elements that can be objectively assessed in published studies, and user testing shows that it can be successfully applied to studies of drugs, medical devices, and clinical and surgical interventions. Although no scoring is provided, study reports that rate relatively well across checklist items merit in-depth examination to understand applicability, effect size, and likelihood of residual bias. The current testing and validation efforts did not achieve clear discrimination between studies fit for purpose and those not, but we have identified a critical, though remediable, limitation in our approach. Not specifying a specific granular decision for evaluation, or not identifying a single study objective in reports that included more than one, left reviewers with too broad an assessment challenge. We believe that future efforts will be more successful if reviewers are asked to focus on a specific objective or question. Despite the challenges encountered in this testing, an agreed upon set of assessment elements, checklists, or score cards is critical for the maturation of this field. Substantial resources will be expended on studies of real-world effectiveness, and if the rigor of these observational assessments cannot be assessed, then the impact of the studies will be suboptimal. Similarly, agreement on key elements of quality will ensure that budgets are appropriately directed toward those elements. Given the importance of this task and the lessons learned from these extensive efforts at validation and user testing, we are optimistic about the potential for improved assessments that can be used for diverse situations by people with a wide range of experience and training. Future testing would benefit by directing reviewers to address a single, granular research question, which would avoid problems that arose by using the checklist to evaluate multiple objectives, by using other types of validation test sets, and by employing further multivariate analysis to see if any combination or sequence of item responses has particularly high predictive validity.

Checklist to operationalize measurement characteristics of patient-reported outcome measures

The GRACE checklist for rating the quality of observational studies of comparative effectiveness: a tale of hope and caution

An Evidence-Based Approach to Choosing Outcome Measures: A Checklist for the Critical Appraisal of Validity, Reliability and Responsiveness Studies

Selecting patient-reported outcome measures: "what" and "for whom"

Interviewing to develop Patient-Reported Outcome (PRO) measures for clinical research: eliciting patients’ experience

The case for an international patient-reported outcomes measurement information system (PROMIS®) initiative

Patient-reported outcome measures in dysphagia: a systematic review of instrument development and validation

Patient-Reported Outcome Measure Development and Validation: A Primer for Clinicians

Measurement Properties of Existing Patient-Reported Outcome Measures on Medication Adherence: Systematic Review

Refining scores based on patient reported outcomes – statistical and medical perspectives

Patient‐Reported Experience Measures in Adult Inpatient Settings: A Systematic Review

Comparable Real-World Patient-Reported Outcomes Data Across Health Conditions, Settings, and Countries: the PROMIS International Collaboration

Patient-Reported Outcome Measures for Swallowing: A Proposed Checklist

A composite measure for patient-reported outcomes in orthopedic care: design principles and validity checks

Patient-reported outcome instruments that evaluate adherence behaviours in adults with asthma: a systematic review of measurement properties

A single patient reported outcome measure for acquired brain injury, multiple sclerosis & Parkinson’s disease

Administering selected subscales of patient-reported outcome questionnaires to reduce patient burden and increase relevance: a position statement on a modular approach

Psychometric evaluation of patient-reported experience measures for peri-anesthesia care: A systematic review based on COSMIN guidelines

The PU-PROM: A patient-reported outcome measure for peptic ulcer disease

The Patient-Perceived Helpfulness of Measures Scale: Development and Validation of a Scale to Assess the Helpfulness of Using Measures in Psychological Treatment

Measurement properties of patient-reported outcome measures of mental help-seeking attitude: a systematic review of psychometric properties