Abstract:Objective To review the findings of studies that have evaluated the design and/or usability of key risk of bias (RoB) tools for the assessment of RoB in primary studies, as categorised by the LATITUDES Network (a searchable library of RoB tools for evidence synthesis): PROBAST, RoB2, ROBINS-I, QUADAS-2, QUADAS-C, QUAPAS, ROBINS-E, and the COSMIN RoB checklist. Study design and setting Systematic review of methodological studies. We conducted a forward citation search from the primary report of each tool, to identify primary studies aiming to evaluate the design and/or usability of the tool. Two reviewers assessed studies for inclusion. We extracted tool features into Microsoft Word and used NVivo for document analysis, comprising a mix of deductive and inductive approaches. We summarised findings within each tool and explored common findings across tools. Results We identified 13 tool evaluations meeting our inclusion criteria: PROBAST (3); RoB2 (3); ROBINS-I (4); QUADAS-2 (3). We identified no evaluations for the other tools. Evaluations varied in clinical topic area, methodology, approach to bias assessment and tool user background. Some had limitations affecting generalisability. We identified common findings across tools for 6/14 themes: 1) challenging items (e.g. RoB2/ROBINS-I "deviations from intended interventions" domain), 2) overall RoB judgement (concerns with overall risk calculation in PROBAST/ROBINS-I), 3) tool usability (concerns about complexity), 4) time to complete tool (varying demands on time e.g. depending on number of outcomes assessed), 5) user agreement (varied across tools), and 6) recommendations for future use (e.g. piloting) and development (add intermediate domain answer to QUADAS-2/PROBAST; provide clearer guidance for all tools). Of the other eight themes, seven only had findings for the QUADAS-2 tool, limiting comparison across tools, and one ("re-organisation of questions") had no findings. Conclusion Evaluations of key RoB tools have posited common challenges and recommendations for tool use and development. These findings may be helpful to people using or developing RoB tools. Guidance is necessary to support the design and implementation of future RoB tool evaluations.

The ROBINS‐I and the NOS had similar reliability but differed in applicability: A random sampling observational studies of systematic reviews/meta‐analysis

Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments

[Series of Risk of Bias Assessment (5): Risk of Bias in Non-randomized Studies-of Interventions (ROBINS-I)].

Assessing risk of bias in a non‐randomized study

[Introduction of a Tool to Assess Risk of Bias in Non-randomized Studies-of Environmental Exposure (ROBINS-E)].

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

Reliability of the risk of bias assessment in randomized controlled trials for nursing: A cross‐sectional study

The risk of bias in observational studies of exposures (ROBINS-E) tool: concerns arising from application to observational studies of exposures

The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomised Studies in Meta-Analyses

Use and Reporting of Risk of Bias Tools in 825 Systematic Reviews of Acupuncture: a Cross-Sectional Study.

ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis

Quality and risk of bias appraisals of systematic reviews are inconsistent across reviewers and centers

Common challenges and suggestions for risk of bias tool development: a systematic review of methodological studies

Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis

A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E)

[Risk on Bias Assessment: (2) Revised Cochrane Risk of Bias Tool for Individually Randomized, Parallel Group Trials (Rob2.0)].

Bias Risk Assessment: (3) Revised Cochrane Bias Risk Assessment Tool for Individual Randomized,cross-over Trials (Rob2.0)

A systematic review: tools for assessing methodological quality of human observational studies

[Introduction of a Tool to Assess Risk of Bias in Non-randomized Studies-of Exposure (2022)].

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355: i4919

Risk of bias tool in systematic reviews/meta-analyses of acupuncture in Chinese journals