Abstract:Background: Different tools exist for assessing risk of bias of intervention studies for systematic reviews. We present a tool for assessing risk of bias across both randomized and non-randomized study designs. The tool was developed by the Evidence Project, which conducts systematic reviews and meta-analyses of behavioral interventions for HIV in low- and middle-income countries. Methods: We present the eight items of the tool and describe considerations for each and for the tool as a whole. We then evaluate reliability of the tool by presenting inter-rater reliability for 125 selected studies from seven published reviews, calculating a kappa for each individual item and a weighted kappa for the total count of items. Results: The tool includes eight items, each of which is rated as being present (yes) or not present (no) and, for some items, not applicable or not reported. The items include (1) cohort, (2) control or comparison group, (3) pre-post intervention data, (4) random assignment of participants to the intervention, (5) random selection of participants for assessment, (6) follow-up rate of 80% or more, (7) comparison groups equivalent on sociodemographics, and (8) comparison groups equivalent at baseline on outcome measures. Together, items (1)-(3) summarize the study design, while the remaining items consider other common elements of study rigor. Inter-rater reliability was moderate to substantial for all items, ranging from 0.41 to 0.80 (median κ = 0.66). Agreement between raters on the total count of items endorsed was also substantial (κw = 0.66). Conclusions: Strengths of the tool include its applicability to a range of study designs, from randomized trials to various types of observational and quasi-experimental studies. It is relatively easy to use and interpret and can be applied to a range of review topics without adaptation, facilitating comparability across reviews. Limitations include the lack of potentially relevant items measured in other tools and potential threats to validity of some items. To date, the tool has been applied in over 30 reviews. We believe it is a practical option for assessing risk of bias in systematic reviews of interventions that include a range of study designs.

Risk of Bias Assessment: (9) Application of the Risk of Bias Assessment Results

Risk of Bias Assessment: (1) Overview

[Introduction on 'assessing the Risk of Bias of Individual Studies' in Systematic Review of Health-Care Intervention Programs Revised by the Agency for Healthcare Research and Quality].

Risk of Bias Assessment: (8) Risk of Bias in Systematic Review (ROBIS)

[Risk on Bias Assessment: (6) A Revised Tool for the Quality Assessment on Diagnostic Accuracy Studies (QUADAS-2)].

[Series of Risk of Bias Assessment (5): Risk of Bias in Non-randomized Studies-of Interventions (ROBINS-I)].

Quality versus Risk-of-Bias assessment in clinical research

Bias Risk Assessment: (3) Revised Cochrane Bias Risk Assessment Tool for Individual Randomized,cross-over Trials (Rob2.0)

[Risk of Bias Assessment: (7) Assessing Bias in Studies of Prognostic Factors].

[Introduction of a Tool to Assess Risk of Bias in Non-randomized Studies-of Environmental Exposure (ROBINS-E)].

[Risk on Bias Assessment: (2) Revised Cochrane Risk of Bias Tool for Individually Randomized, Parallel Group Trials (Rob2.0)].

[Risk Related to Bias Assessment: (4) Revised Cochrane Risk of Bias Tool for Cluster-Randomized Control Trials (Rob2.0)].

A case study of the informative value of risk of bias and reporting quality assessments for systematic reviews

Risk of bias assessments in systematic reviews and meta-analyses of behavioral interventions for substance use outcomes

Towards the automatic risk of bias assessment on randomized controlled trials: A comparison of RobotReviewer and humans

Automating risk of bias assessment for clinical trials

Tools to Assess Risk of Bias in Systematic Reviews of Nursing Intervention in China: Global Implications of the Findings.

[Introduction of a Tool to Assess Risk of Bias in Non-randomized Studies-of Exposure (2022)].

Meta‐epidemiological Study: a Step‐by‐step Approach by Using R

The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies

RoB 2: a revised tool for assessing risk of bias in randomised trials